XML Sitemap QA: URLs, Canonicals, Lastmod, Status, Splits

XML sitemap QA compares a generated sitemap with the site’s approved canonical URL inventory and live public behavior. It verifies valid XML, absolute in-scope URLs, successful indexable responses, consistent canonicals, truthful modification dates, complete and nonoverlapping splits, discoverability, and Search Console processing evidence.

A sitemap helps search engines discover preferred URLs, but submission is a hint rather than a guarantee of crawling or indexing. Never include private, preview, account, search-result, filtered, duplicate, or legally restricted URLs merely to make a coverage count look larger.

Create A Canonical URL Inventory

Export the public routes that the organization intends search engines to discover. Record page type, source record, expected canonical, indexability, language, publication state, meaningful modification time, and owning sitemap shard. Date and hash the inventory used for the release.

Do not build the expected list by copying the sitemap under test. Derive it from authoritative route and content data, then compare both directions. This reveals missing URLs as well as unexpected inclusions.

Locate Every Sitemap Entry Point

Check robots.txt declarations, submitted Search Console paths, the sitemap index, child sitemaps, CMS settings, framework routes, CDN rewrites, and legacy locations. Record redirects and final response URLs. Remove stale references only through an approved migration plan.

A file hosted at the site root can describe URLs across the host more naturally. Google notes that sitemap location affects URL scope unless the file is submitted through Search Console. Keep ownership and cross-site submission requirements explicit.

Verify The HTTP Response

Request each sitemap while signed out and capture final URL, redirect chain, status, content type, encoding, compression, cache headers, body size, and body hash. Require a successful stable response without authentication, HTML error templates, challenge pages, or environment-specific hosts.

Test conditional requests and cache expiry so a deployment does not serve an older shard indefinitely. Confirm a missing child returns an honest error rather than a branded 200 page that a parser may reject ambiguously.

Parse XML Strictly

Use a namespace-aware XML parser with external entity and network access disabled. Validate well-formed structure, one supported root, required loc elements, recognized optional elements, character escaping, and UTF-8 encoding. Reject truncated bodies, duplicate roots, HTML prefixes, and unsafe declarations.

Do not validate with regex or a browser’s pretty view. Store parser errors with line and column, but avoid logging an enormous body. Test both compressed and uncompressed responses when the deployment uses gzip.

Check Protocol Limits

The Sitemaps protocol and Google documentation limit one sitemap to 50,000 URLs and 50MB uncompressed. Measure the actual decoded XML bytes and URL count, leaving operational headroom for growth. Split before the boundary rather than relying on an exact-edge deployment.

Apply the same 50,000-entry and 50MB uncompressed limits to sitemap index files according to the protocol. Verify compressed transfer size does not conceal an oversized uncompressed document.

Require Absolute URLs

Every loc should be a fully qualified URL with approved HTTPS scheme, lowercase registered host, valid port policy, normalized path, and properly escaped query string. Google recommends absolute URLs because crawlers request what is listed.

Reject relative paths, protocol-relative URLs, CMS or staging hosts, localhost, IP addresses, credentials, fragments, malformed percent encoding, and whitespace. Normalize only for comparison; do not silently publish a rewritten URL without fixing the generator source.

Enforce Host And Scope

Confirm each URL belongs to the verified site and the sitemap’s allowed scope. If cross-site submission is intentionally configured, verify ownership and the documented hosting arrangement for every host. Separate international properties where operational ownership requires it.

Audit scheme and host variants such as HTTP, www, apex, alternate subdomains, and trailing-dot hosts. A consistent sitemap should reinforce the chosen public architecture rather than expose every reachable variant.

Compare Canonicals

Fetch representative and risk-based pages, read the final URL and rel canonical, and compare them with the sitemap loc. Google advises listing preferred canonical URLs. Sitemap inclusion is a canonical hint, not a substitute for redirects and consistent page annotations.

Block URLs that redirect, canonicalize elsewhere, or conflict across HTML, HTTP headers, hreflang, internal links, and sitemap entries. Use rendering and canonical QA when JavaScript can alter page metadata.

Verify Indexability

For every URL or a complete automated set, check status, robots meta, X-Robots-Tag, authentication, content type, and canonical. Exclude noindex, blocked, private, soft-404, error, and non-HTML resources unless a supported specialized sitemap intentionally includes them.

Robots.txt blocking does not make a poor sitemap entry acceptable. A blocked URL can still create conflicting discovery evidence, and crawlers may be unable to inspect its page-level directives. Align sitemap inclusion with the intended indexable set.

Remove Redirects And Errors

Request URLs without following redirects first, then record the final destination. Flag 3xx, 4xx, 5xx, timeout, redirect loops, TLS failure, and intermittent errors. Replace old URLs with their final canonical destinations after the migration policy is approved.

Sample from every shard continuously, but run a complete response audit before major submission or migration when feasible. A sitemap that mostly works can still waste crawl attention on thousands of predictable failures.

Detect Duplicates

Canonicalize only for analysis and find exact duplicate loc values, case variants, default ports, slash variants, parameter permutations, encoded equivalents, and URLs repeated across shards. Investigate why each variant exists before choosing the approved form.

Do not deduplicate by deleting rows from generated XML after the fact. Fix route ownership, query policy, canonical selection, and generator inputs so the same defect does not return on the next build.

Validate Lastmod

Use lastmod only when the source can report the last significant page modification truthfully. Follow supported date formats. Google says useful values should match reality and represent meaningful changes, such as primary content, structured data, or important links.

Reject build time, sitemap generation time, request time, and blanket current dates applied to unchanged pages. Compare samples against CMS revisions, repository history, or authoritative records. It is acceptable to omit lastmod when confidence is low.

Ignore Priority And Changefreq Myths

Do not spend QA effort tuning values that the target search engine does not use. Focus on correct canonical URLs and truthful modification evidence. Keep optional protocol elements only when a documented consumer and maintenance owner justify them.

Remove automatically generated values that imply precision without evidence. A daily changefreq on a static page does not make it recrawl daily, and a high priority does not create ranking authority.

Audit Sitemap Splits

Choose stable shards by content type, locale, or another operationally meaningful boundary. Verify each expected URL appears exactly once, every child is reachable, counts sum correctly, shard names remain stable, and empty or retired shards follow a documented lifecycle.

Stable splits make Search Console processing and incident diagnosis easier. Avoid random chunking that moves thousands of unchanged URLs between files on every build and makes diffs noisy.

Validate The Sitemap Index

Parse the index separately and require absolute child sitemap URLs, approved host and scope, unique entries, successful child responses, correct lastmod semantics, and complete coverage. Detect indexes that reference another index if the target consumer or local policy does not support that pattern.

Compare the deployed index with the release manifest. A child produced successfully but omitted from the index is invisible to that discovery path, while a stale child can continue advertising retired URLs.

Check Specialized Extensions

For image, video, news, or alternate-language annotations, validate the relevant namespace, required fields, ownership, page relationship, and public accessibility. Do not add extension markup merely because a generator supports it.

Compare image and video URLs with visible page content and canonical ownership. Remove expired news entries and inaccessible media according to the applicable official rules. Keep extension failures isolated so they do not corrupt the base sitemap.

Submit And Read Processing Evidence

Reference the sitemap in robots.txt or submit it through Search Console according to the operating plan. Google deprecated the unauthenticated ping endpoint, so remove jobs that depend on it. Record submission time, discovered type, last read, status, and reported errors.

Submission does not guarantee indexing. Compare submitted and indexed patterns by shard, then investigate page quality, canonical conflicts, discovery, rendering, or crawl issues. Do not repeatedly resubmit an unchanged valid sitemap as a substitute for diagnosis.

Monitor Every Release

After content and route deployments, regenerate from authoritative data and diff URL additions, removals, moves, lastmod changes, shard movement, and body hashes. Alert on count drops, foreign hosts, parser failures, stale generation, elevated redirects, or error responses.

Use the broader SEO release gate to connect sitemap evidence with routes, metadata, structured data, and monitoring. The Search Engine Optimization course can build the judgment behind these checks.

FAQ

Does submitting a sitemap guarantee indexing?

No. It is a discovery and canonical hint; search engines still decide whether and when to crawl and index each URL.

Should lastmod always be included?

Only when it can reflect the page’s last significant modification truthfully in a supported format.

Can a sitemap list redirecting URLs?

It should list the final preferred canonical URLs, not redirects or alternate variants.