robots.txt

A file at a site’s root telling crawlers where they may and may not go. A stray Disallow can wall off pages you wanted found.

First used in: 1.1 · Get found, get measured