AI crawler

An AI crawler is a bot that fetches web pages on behalf of an AI company. The same company often runs several for different jobs — gathering training data, powering live search retrieval, and fetching a specific page a user acts on — and each has its own user-agent name you can allow or block in robots.txt.

First used in: 3.3 · AI crawler access