Technical SEO I — crawl & index control

IntermediateDuration ~20 min video + 90 min hands-onTools Screaming Frog (free tier), Google Search Console, A browser + view-source

Why this lesson

Everything you’ve done so far — keywords, on-page, content — assumes search engines can actually reach your pages and are willing to keep them in the index. If they can’t crawl a page, or you’ve accidentally told them not to index it, none of the rest matters. Technical SEO at this level is about control: deciding what gets crawled, what gets indexed, and which version of a page is the canonical one. Get it right and your best pages get found and your junk pages stay out of the way. Get it wrong and you can quietly delete your whole site from Google with one line of code. As Sam Oh puts it, if pages aren’t getting indexed, nothing else really matters.

Watch

Watch for: The four indexing controls and exactly what each does: the noindex meta tag (keeps a page out of the index — and how a forgotten staging noindex silently deletes pages after a redesign), robots.txt directives (User-agent + Disallow, the difference between blocking a crawler and hiding from the index), XML sitemaps (listing your important URLs), 301 redirects (consolidating signals from an old URL to a new one), and the canonical tag (naming the preferred version to solve duplicate content).

Segment: full video — using site: search operators to surface crawl/index issueswatch full video

Watch for: How to find crawl and index problems with nothing but Google search operators: site:domain.com to see what's indexed, -inurl:https to find unsecure indexed pages, intitle: to catch wrong titles from a broken template, and inurl:page / inurl:sort to expose pagination and faceted-navigation URLs that shouldn't be indexed. Note where he flags a crawler like Screaming Frog does it faster and finds 301s and 404s Google won't show you.

The explainer

Crawl and index are two separate gates. A search engine first crawls a URL (fetches it) and then decides whether to index it (store it for ranking). You control each gate with a different tool, and mixing them up is the classic mistake. robots.txt — a file at your root with User-agent and Disallow directives — controls crawling: it says which URLs a bot may fetch. A noindex meta tag controls indexing: it lets the bot fetch the page but tells it not to keep it. The trap: if you Disallow a page in robots.txt, the crawler never fetches it, so it never sees a noindex tag on it — the two can’t be combined to remove a page. To reliably drop a page, allow crawling and use noindex.

Canonicals and duplicate content. The web is full of the same content on multiple URLs — http vs https, www vs non-www, tracking parameters, printer versions. That’s duplicate content, and it splits signals across URLs and confuses engines about which to rank. The fix is canonicalization: pick one preferred URL and mark the duplicates with a rel="canonical" tag pointing at it, so links and ranking signals consolidate onto the canonical version. Note that a canonical is a strong hint, not a command — Google can override it (for example, it’ll usually pick the https version even if you canonicalize to http). When you can, a 301 redirect is stronger than a canonical because it physically sends everyone to one URL.

Sitemaps and crawl budget. An XML sitemap lists the important URLs on your site so engines can discover them intelligently; you submit it in Search Console and reference it in robots.txt. On a large site this matters because of crawl budget — the finite amount of crawling a search engine will spend on your site per period. If bots burn that budget on thousands of junk URLs, your important pages get crawled less often and fresh content takes longer to appear. Keeping crawlers away from low-value URLs is as much about protecting crawl budget as about indexing.

Faceted navigation: where budget goes to die. Filterable stores and listings are the biggest crawl-budget sink. Faceted navigation — colour, size, price, sort filters — multiplies into a combinatorial explosion of URLs like /shoes?color=red&size=10&sort=price, each a near-duplicate of the others. Left unchecked, engines crawl and index thousands of these thin variants. You decide which facets deserve indexing (maybe color), and for the rest you canonicalize back to the clean category, block the parameters, or noindex them.

Read the status codes. Every URL a crawler hits returns an HTTP status code, and reading them is core diagnostic literacy. 200 = OK. 301 = permanent redirect (moved for good, passes signals — use this for consolidation). 302 = temporary redirect (moved for now, keep the old URL). 404 = not found (page is gone, no forwarding). 410 = gone (deliberately, permanently removed — a stronger “don’t come back” than 404). 503 = service unavailable (temporary; the right code to serve during planned downtime so engines retry instead of dropping you). A crawl report is mostly a table of these codes; the outliers — unexpected 404s, redirect chains, stray 302s that should be 301s — are your fix list.

Find the issues. Two levels of tooling. Free and fast: Google itself, via site: search operators — site:yourdomain.com shows what’s indexed, -inurl:https finds unsecure indexed pages, inurl:sort exposes faceted junk. Thorough: a crawler like Screaming Frog fetches every URL and hands you status codes, duplicate titles, redirect chains, and canonical mismatches in one table — including the 301s and 404s Google won’t surface for you. For the deepest view, log-file analysis reads your server logs to see what search engine bots actually crawled and how often — the ground truth about where your crawl budget is really going.

Do it

1. Open yourdomain.com/robots.txt on your practice site. Read every Disallow. Confirm nothing important is blocked, and that your sitemap is referenced.
2. Run site:yourdomain.com in Google. Roughly how many pages are indexed? Now run -inurl:https and inurl:sort (or another parameter) — flag anything indexed that shouldn’t be.
3. Crawl the site with Screaming Frog. Sort by status code: list every non-200 (301, 302, 404, 410). Note redirect chains and any 302 that should be a 301.
4. Find duplicate content: sort by title and by canonical tag. Where duplicates exist, decide the canonical URL for each set.
5. If the site has filters, map its faceted navigation: which parameters create URLs, and which you’d index vs canonicalize vs block.
6. Write a prioritized crawl-and-index fix list in the Level 2 workbook.

Level 2 workbook — crawl/index control checklist, status-code cheat sheet & faceted-nav decision gridlevel-2-workbook.pdf116 KBOriginal course material — free to use

Terms introduced

Terms introduced in this lesson

• Crawl budget
• Canonicalization
• Duplicate content
• Faceted navigation
• Status codes
• Log-file analysis

Each term links to the glossary, where it is defined by the way you just used it.

Check

Check yourself

1. What is the difference between a robots.txt Disallow and a noindex meta tag?

   Disallow tells crawlers not to crawl a URL; noindex lets them crawl it but tells them not to keep it in the index — and a page blocked by robots.txt can never see its noindex tagnoindex blocks crawling; Disallow removes from indexBoth remove a page from the index instantlyThey do exactly the same thing

2. A faceted-navigation store generates URLs like /shoes?color=red&size=10&sort=price. Why is this a technical SEO problem?

   It looks uglyIt is never a problemGoogle bans all query parametersFilter combinations create thousands of near-duplicate, low-value URLs that waste crawl budget and cause duplicate content — you must decide which to index and canonicalize or block the rest

3. You have permanently moved a page to a new URL. Which status code should the old URL return?

   302 Found404 Not Found503 Service Unavailable301 Moved Permanently — it redirects users and passes the old URL's signals to the new one

Check answers

You can move on when you can… explain the difference between crawling and indexing, choose the right control (robots.txt vs noindex vs canonical vs 301) for a given goal, read the common status codes, and identify where faceted navigation is wasting crawl budget on a real site.

Mark this lesson complete

Go deeper

• Google Search Central — “Crawling and indexing” documentation: the authoritative reference for robots.txt, canonicalization, sitemaps and status codes, straight from the source.
• Next: 2.4 · Technical SEO II — rendering, speed & mobile — once a page can be crawled, make sure engines can actually render, load, and read it fast.