Crawling & Indexing

Definition

Crawling is how search engines find your pages by following links and reading their content. Indexing is the step that follows: storing those pages in the search engine's database so they're eligible to appear in results. A page that isn't indexed can't rank for anything.

In depth

First a crawler visits a page and reads it, often guided by your XML sitemap and internal linking. Then the engine decides whether the page is worth storing in its index. Only indexed pages can show up in results, so crawling and indexing are the gate every other piece of technical SEO has to pass through.

This is the foundation for a contractor's site: it doesn't matter how good your kitchen remodeling page is if search engines never indexed it. Confirming that your money pages are actually in the index is the first thing worth checking, because no organic traffic can arrive until they are.

The common trap is accidentally blocking pages from being indexed, often a leftover 'noindex' tag from a staging site or an overzealous robots rule, so good pages stay invisible. We verify index status in Google Search Console, clear blockers, and make sure the pages you depend on are eligible to rank before tuning anything else.

Worked example

Related terms