Glossary · Glossary

Crawling

Crawling is the process search systems use to fetch pages, discover links, and inspect web content.

Crawling is the process search systems use to fetch URLs, inspect their responses, parse links, and discover additional pages. A crawler starts with known URLs from links, sitemaps, previous crawls, or submitted URLs, then follows discoverable paths across the site.

For a static content site, crawling is the discovery step that turns a route into a candidate search source. It is where search systems learn that pages such as glossary entries, field guides, reports, and comparison pages exist.

Why it matters

GEO still needs discoverable source material. AI-influenced search features can surface links and cited sources, but those sources usually need to be accessible, indexable, and understandable first. If a page is never crawled, it has a much weaker chance of becoming a visible search result or supporting source.

Crawling also helps teams diagnose broken publication paths. If a new glossary page is listed in the sitemap but has no internal links, or if it redirects through an unexpected chain, crawl data can reveal the problem before it becomes a long-term visibility issue.

How it differs

Crawlability asks whether the crawler can access a URL. Crawling is the actual fetch and discovery process. Indexing comes after crawling, when search systems decide whether to store and organize the content for possible retrieval.

Ranking is separate again. Crawling makes a page known; indexing makes it eligible; ranking orders eligible results for a query.

Example flow

Glossary index
  -> /glossary/crawling/
       -> related link: /glossary/crawlability/
       -> related link: /glossary/indexing/
       -> related guide: /guides/geo-vs-seo/

That link path helps a crawler move from the index into the term page, then into adjacent concepts. If the page is only present in sitemap.xml, discovery can still happen, but it is weaker for users and less useful as a topical path.

How teams use it

Teams support crawling by keeping important pages linked, avoiding broken redirects, serving successful responses, keeping important content public, and making route changes deliberately. For technical SEO reviews, the key questions are:

Common misunderstanding

Crawling is not proof of indexing. A crawler can fetch a page and search systems can still choose not to index it because it is duplicate, blocked by a page-level directive, low value, canonicalized elsewhere, or not useful enough for search results.