Many sites build part or all of their content in the browser with JavaScript. The raw HTML the server returns may contain little or none of it.
A crawler that renders JavaScript loads the page in a browser engine, waits for scripts to run and then reads the result.
If important content or links only exist after rendering, crawlers that do not render, including some AI crawlers, may never see them.
With a Marketer licence and a local Chrome or Chromium, VisibilityKit Crawler renders pages and compares the result with the raw response. Rendering uses a bounded delay, and a failed render is recorded as a failure rather than counted as a successful check.
Google does, often with a delay. Many other crawlers, including several AI crawlers, read only the raw HTML.
Marketer and Agency. The free tier audits the raw HTML.
The difference between the HTML a server sends and the page after JavaScript has run.
The process by which AI platforms discover and index web content for use in generating responses.
Whether a page is able to be included in a search engine's index.
A systematic check of a website's crawlability, indexability, links, redirects and on-page elements.
A desktop SEO crawler for macOS, Windows and Linux. Find what is broken, assign the work, then verify every fix. Crawl data stays on your machine.
Get Crawler free