Guide

Sitemap audit: compare your sitemap with a crawl

A sitemap should list only the URLs you want indexed: final, 200, indexable and canonical. After a few releases most sitemaps drift. Comparing the sitemap with a fresh crawl export shows exactly which entries to drop, replace or add.

What belongs in a sitemap

Search engines treat a sitemap as a list of preferred URLs. Each entry should return 200, should not carry a noindex, should be its own canonical, and should not be blocked by robots.txt. Every entry that breaks one of these rules sends a mixed signal and wastes crawl requests.

Get the two inputs

You need the sitemap URL list and a crawl export of the same site from your crawler, with at least the URL and status code, and ideally noindex, robots-blocked, canonical and final URL columns. Crawl soon after the latest release so both describe the same site.

One verdict per URL

Give each sitemap URL a single verdict, checked in a fixed order so results are repeatable: not crawled at all, 404 or 410, 5xx, redirected (3xx), noindex, canonicalized to another URL, or blocked by robots.txt. A URL that passes all of them is fine. A fixed order matters because one URL often fails several rules at once, and you want one clear action for it.

Replace, do not just delete

A redirected sitemap URL usually has a good final URL; a canonicalized one points at the page you actually want indexed. Replace the entry with that target when the target itself is indexable, rather than simply removing it. 404/410 entries are removed. 5xx entries need a server fix first, then a recrawl.

Find the orphans

The other direction matters too: indexable pages the crawl found that are not in the sitemap. These are often new pages a release added without updating the sitemap generator. List them and add the ones you want indexed.

Compare runs over time

A one-off audit drifts again. Keep the previous run's URL and verdict list, and after the next release compare: which URLs were resolved, which are new problems, and which regressed. A sorted, deduplicated output with a checksum makes it easy to see whether the sitemap actually changed.

Common pitfalls

Trailing slash and protocol variants (http vs https, /page vs /page/) are different URLs; the sitemap should use the canonical form.
Staging or parameter URLs sometimes leak into generated sitemaps.
A robots.txt block does not remove a URL from a sitemap; fix both together.

Check it automatically

Sitemap Crawl Reconciler reads your sitemap URL list and a crawl CSV (url and status, optionally noindex, robots_blocked, canonical, final_url and gsc_state) and gives each sitemap URL one verdict by fixed priority: ok, not_crawled, status_404_410, status_5xx, redirect_3xx, noindex, canonical_away or robots_blocked. It suggests replacements, lists indexable pages missing from the sitemap, returns a cleaned sorted list with its SHA256, and diffs against a previous run. It makes no network requests. Three checks are free; the monthly plan is $12.