An XML sitemap lists canonical, indexable URLs, optionally with a last-modified date. Large sites split it into several files referenced from a sitemap index.
Sitemaps help discovery but do not guarantee indexing.
Sitemaps help search engines find new and deep pages faster. A sitemap full of redirects, 404s or noindexed pages sends mixed signals.
VisibilityKit Crawler can export a sitemap from a crawl, and comparing a crawl with your sitemap URLs surfaces orphan pages and sitemap entries that no longer return 200.
Only canonical URLs that return 200 and are meant to be indexed.
It is optional if every page is well linked, but it costs little and helps new pages get found.
A text file at the root of a site that tells crawlers which paths they may and may not fetch.
A page that exists on your site but has no internal links pointing to it.
Whether a page is able to be included in a search engine's index.
An HTML link element that tells search engines which URL is the preferred version of a page.
Crawling only a supplied list of URLs, without following their links.
A desktop SEO crawler for macOS, Windows and Linux. Find what is broken, assign the work, then verify every fix. Crawl data stays on your machine.
Get Crawler free