How search engines (and answer engines) work
Why this lesson
Section titled “Why this lesson”You cannot optimize a system you can’t picture. Almost every SEO decision you’ll ever make is really an answer to one of three questions: can the engine reach my page, can it understand and store it, and does it judge it the best answer? Those three questions map exactly to the three stages every result travels through — crawl, index, rank. Once that pipeline is in your head, “technical SEO,” “on-page,” and “links” stop being disconnected topics and become interventions at specific stages. Then we add the 2026 layer: what an answer engine does after ranking, because that’s a different pipeline with its own leverage points.
Segment: 4:30–8:20 — How Google crawls, indexes & rankswatch full video
The explainer
Section titled “The explainer”The classic pipeline: crawl → index → rank
Section titled “The classic pipeline: crawl → index → rank”Crawl. Search engines run automated programs — crawlers, also called spiders or bots (Google’s is Googlebot) — that move through the web by following links. They begin from a set of already-known URLs (seeds), fetch each page, and follow the hyperlinks on it to discover new pages, over and over. The practical consequence for you: if nothing links to a page and it’s not in a sitemap, it may never be found. Crawlability is stage one, and a page that can’t be crawled cannot rank for anything, no matter how good it is.
Index. A crawled page gets processed and stored in the index — a colossal database of the web’s content. This is not passive filing; the engine tries to understand the page: what it’s about, what entities it mentions, whether it’s a duplicate of something already stored. A page can be crawled but not indexed (thin, duplicate, or blocked by a noindex instruction), and a page that isn’t indexed is invisible to search. When you hear “get your page indexed,” this is the stage being talked about.
Rank. When someone searches, the engine pulls every candidate page from the index and orders them for that specific query using hundreds of signals. Nobody outside Google knows the exact recipe — and Google retunes it hundreds of times a year — but three fundamentals do most of the explaining, and you’ll spend Level 1 on them: backlinks (links from other sites, which act like votes of trust), search intent (does the page match the kind of thing the searcher wanted — a recipe, a product, a definition), and content depth (does it actually answer the question, fully). Ranking is where “the best answer wins” — but only among pages that made it through crawl and index first.
Hold onto the dependency chain, because it’s the backbone of the whole course: crawl enables index enables rank. Most catastrophic SEO failures are not “we ranked #6 instead of #3” — they’re a page silently stuck at stage one or two, invisible, while its owner tweaks stage three.
The new layer: how answer engines retrieve and cite
Section titled “The new layer: how answer engines retrieve and cite”Modern AI answers add a second pipeline that runs on top of the classic one. When you ask ChatGPT (with search), Perplexity, Gemini, or a Google AI Overview a question, the system broadly does this: it retrieves a set of pages relevant to your question — usually from an existing search index (ChatGPT leans on Bing’s, AI Overviews on Google’s) — then generates a written answer that is grounded in those retrieved pages, and cites some of them as sources. This retrieve-then-generate pattern has a name you’ll meet again: RAG, retrieval-augmented generation.
Two things follow immediately, and they define all of Level 3:
-
You still have to be retrievable. The AI can only ground its answer in pages it could crawl and that sit in the index it draws from. So every classic SEO fundamental is a precondition for GEO — plus one sharp new one: most AI crawlers do not run JavaScript, so content that only appears after a page’s scripts execute can be invisible to them even when it’s fine for Google.
-
Ranking and citing are different competitions. Being the top-ranked blue link and being the source the AI quotes are decided differently. The AI favors content it can lift a clean, self-contained, well-supported statement from — which is why a page ranked #7 can end up cited in the answer while #1 is ignored. That gap is the entire opportunity of AEO and GEO, and we’ll turn it into a repeatable method later.
For now, the whole picture in one line: a page is crawled, indexed, and ranked — and then, increasingly, an answer engine retrieves it, and either cites it or doesn’t. Four stages. Every technique in this course improves your standing at one of them.
- Pick any page you control or know well (a site, a blog post, even a social profile). Ask, honestly, at which stage it could be failing: Can it be crawled? Is it indexed? Does it rank? Does any AI cite it?
- Do a quick index check: search Google for
site:yourdomain.com— the results are (roughly) what Google has indexed for that site. Few or zero results is a stage-one/two problem, not a ranking problem. - Ask an AI answer engine (ChatGPT with search, or Perplexity) a question your page should be able to answer. Does it cite you? Someone else? Note who.
- Write the four stages — crawl, index, rank, cite — on a sticky note. For the rest of this course, when a tactic appears, place it on the note. You’ll be surprised how cleanly everything sorts.
Terms introduced
Section titled “Terms introduced”Check yourself
Put the classic search pipeline in the right order.
What does a crawler actually do?
When an AI answer engine like Perplexity answers a question, what is it mostly doing under the hood?
You can move on when you can… draw the four stages — crawl → index → rank → cite — from memory, say in one sentence what happens at each, and explain why a page stuck at crawl or index can’t be fixed by working on ranking.
Go deeper
Section titled “Go deeper”- Google Search Central — “In-depth guide to how Google Search works”: the official version of crawl → index → serve, straight from the source.
- Next: 0.3 · Set up your free stack & how this course works — install the free tools that let you see all four stages happening on your own site.