Crawler (spider / bot)
An automated program that moves through the web by following links, fetching copies of pages. Googlebot is Google’s. Also used to mean an SEO auditing tool like Screaming Frog.
First used in: 0.2 · How search engines work
An automated program that moves through the web by following links, fetching copies of pages. Googlebot is Google’s. Also used to mean an SEO auditing tool like Screaming Frog.
First used in: 0.2 · How search engines work