Crawl settings

Crawl modes

Mode	Use it for
Whole site	Auditing everything reachable from a starting URL. Follows internal links.
Single page	One URL, audited in isolation.
URL list	A pasted list of URLs — audited exactly as given, no link-following.

Advanced options

Option	What it does
Max pages	Stops the crawl after this many pages. Also capped by your plan’s per-crawl limit.
Max depth	How many clicks from the start URL to follow. Depth 1 = the start page and everything it links to.
Verify external links	Checks links that point off-site too, so you catch outbound 404s.
Respect robots.txt	On by default. Turn it off only for sites you own — e.g. a staging site that blocks all crawlers.
Render JavaScript	Loads each page in headless Chrome before auditing, for client-side-rendered sites. Slower per page, but sees what a browser (and Google) sees.

Speed and concurrency

There’s no concurrency knob — Crawlie scales the number of parallel fetches to the size of the crawl automatically, within polite bounds. How the crawler behaves toward your server is covered in How the crawler behaves.

Plan caps

Each plan caps pages per crawl — 1,000 on Hobby, 50,000 on Pro, 250,000 on Business, 1,000,000 on Scale. If a site has more pages than your cap, we crawl up to the cap and mark the report as truncated so you know it isn’t the whole picture. Each crawled page also spends one crawl credit.