How the crawler behaves

Identification

Crawlie identifies itself with the crawlie user-agent token on every request. To target it in robots.txt:

User-agent: crawlie
Disallow: /internal/

robots.txt

Respected by default. Before a site crawl we fetch your robots.txt and skip anything it disallows for the crawlie token — skipped URLs are recorded in the report as blocked by robots, so nothing disappears silently.

You can disable this per crawl in Advanced options — meant for sites you own, like a staging environment that blocks all crawlers.

Sitemaps and discovery files

Site crawls look for your sitemap.xml and use it to seed the crawl, so pages that exist but aren’t well linked still get audited. The report also records whether we found a robots.txt, a sitemap (and how many URLs it lists), and an llms.txt — all three are themselves audit signals.

Pacing

Concurrency is scaled to the size of the crawl within fixed bounds and hard-capped, response bodies are size-limited, and every crawl has an overall timeout. If a crawl hits trouble you can cancel it at any point and keep the partial report.

JavaScript rendering

Off by default — most sites don’t need it and plain fetches are faster. Turn on Render JavaScript for client-side-rendered apps; each page is loaded in headless Chrome before the checks run.