XML Sitemap Optimization
A sitemap is a map, not a magnet
An XML sitemap lists the URLs you want search engines to know about, along with optional metadata about each one. It does not pull pages into the index — it increases the odds a URL gets crawled, especially on sites where internal linking alone won't surface everything in a reasonable time. On a small, well-linked site, a sitemap is almost decorative. On a 500,000-page marketplace or a site that just migrated platforms, it's one of the few levers that actually moves discovery speed.
Sitemap index files: sitemaps of sitemaps
A single sitemap file is capped at 50,000 URLs and 50MB uncompressed. Past that, you need a sitemap index file — a small XML file that points to multiple child sitemaps. Large sites split child sitemaps along logical boundaries: by content type (products, categories, blog posts), by date (useful for news sites), or by section. This isn't just a size workaround — it's a diagnostic tool. Submitting sitemap-products.xml separately from sitemap-blog.xml lets you check indexation rates per content type in Search Console instead of getting one blended number for the whole site.
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://example.com/sitemap-products.xml</loc>
<lastmod>2026-10-01</lastmod>
</sitemap>
<sitemap>
<loc>https://example.com/sitemap-categories.xml</loc>
<lastmod>2026-09-28</lastmod>
</sitemap>
<sitemap>
<loc>https://example.com/sitemap-blog.xml</loc>
<lastmod>2026-10-10</lastmod>
</sitemap>
</sitemapindex>priority and changefreq: mostly ignored, still occasionally useful
The <priority> (0.0–1.0) and <changefreq> (e.g. daily, weekly) tags were part of the original sitemap protocol, and Google has said plainly it ignores both when deciding what to crawl. Bing has historically given them a little more weight, but not much. The practical advice: don't spend engineering time hand-tuning priority values across thousands of URLs expecting a crawl boost. The one tag that still earns its place is <lastmod> — Google does use an accurate last-modified timestamp as a signal for whether a URL needs re-crawling, which is far more valuable than priority ever was.
A fake or stale lastmod backfires
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/products/trail-running-shoe</loc>
<lastmod>2026-09-15</lastmod>
</url>
<url>
<loc>https://example.com/products/waterproof-jacket</loc>
<lastmod>2026-10-02</lastmod>
</url>
</urlset>What belongs in a sitemap — and what doesn't
A sitemap should contain canonical, indexable URLs only: no redirects, no 404s, no noindex pages, no duplicate parameterized variants. Including non-canonical or blocked URLs sends mixed signals and wastes the crawler's attention on the exact pages your canonical tags are telling it to ignore. Before submitting, cross-check the sitemap against the canonical tag on each URL — a sitemap full of URLs that canonicalize elsewhere is a common and avoidable audit finding.
Sitemap hygiene: include vs. exclude
| URL type | Include in sitemap? | Why |
|---|---|---|
| Canonical, indexable pages | Yes | This is the sitemap's actual purpose |
| Redirecting URLs (301/302) | No | Wastes a crawl on a hop instead of content |
| noindex pages | No | Contradictory signal — why list what you've excluded |
| Non-canonical duplicates (?sort=, ?ref=) | No | Dilutes focus from the canonical version |
| 404 / 410 pages | No | Should never have been generated into the file |
Submission vs. discovery
Submitting a sitemap through Search Console or referencing it in robots.txt doesn't make Google 'process' it instantly — it gets fetched and parsed on Google's own schedule, and the URLs inside still go through the normal crawl queue. What submission actually buys you is visibility: the Sitemaps report in Search Console tells you how many submitted URLs are indexed versus excluded, which is the fastest way to spot a systemic indexing problem across thousands of pages at once, rather than checking individual URLs one by one.
User-agent: *
Disallow: /checkout/
Sitemap: https://example.com/sitemap-index.xmlDynamic generation beats static files at scale
What's next
Sitemaps tell crawlers what exists; canonical tags tell them which version of what exists is authoritative. The two work together constantly, and most sitemap audits turn into canonical audits halfway through.
Next: Canonical Tags Deep Dive →
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.