The free, open-source Screaming Frog alternative

crawlie vs Screaming Frog at a glance

Is crawlie a good Screaming Frog alternative?

Yes — especially if you want something free, scriptable, and agent-friendly. crawlie runs the same core technical-SEO checks (broken links, redirects, titles & meta, canonicals, indexability, structured data) with no URL cap and no licence, and adds things Screaming Frog doesn't have. crawlie now renders JavaScript too (the `--render` flag drives headless Chrome), so the main remaining trade-off is Screaming Frog's built-in Google Search Console, Analytics, and PageSpeed integrations.

What does crawlie do that Screaming Frog doesn't?

Three things. It's agent-native — an MCP server and Claude Skills let an AI agent run a full audit and read the results. It grades GEO (Generative Engine Optimization): how citable your pages are by ChatGPT, Perplexity, and Google AI Overviews. And every finding ships with a plain-English explanation of why it matters and how to fix it. It's also open source, so you can read exactly what it checks and extend it.

Is crawlie really free, with no URL limit?

Yes. crawlie is MIT-licensed and free for any number of URLs. Screaming Frog is free up to 500 URLs per crawl; beyond that it needs a paid annual licence. For sites larger than 500 pages, crawlie removes that wall entirely.

Where Screaming Frog is still the better choice

No tool wins on everything — here's where Screaming Frog currently leads, so you can choose with eyes open:

• Built-in integrations — it pulls Google Search Console, Analytics, and PageSpeed/Lighthouse data into the crawl. crawlie doesn't connect those yet.
• XPath custom extraction — crawlie now does CSS-selector and regex extraction, but Screaming Frog also supports XPath.
• A mature, refined desktop app on Windows, macOS, and Linux with years of polish.

Frequently asked questions

Is crawlie free?

Yes — crawlie is free and open-source under the MIT licence, with no URL cap.

Does crawlie render JavaScript like Screaming Frog?

Yes. With the `--render` flag, crawlie loads each page in headless Chrome and audits the rendered DOM — so it sees client-rendered (SPA) content, the same as Screaming Frog. It also flags pages whose content only appears after JavaScript runs (the `content-requires-js` check).

Can I run crawlie in CI?

Yes. The CLI outputs clean JSON and sets exit codes (`--fail-on`), so you can gate a build on SEO regressions. There's a ready-to-paste GitHub Actions workflow in the docs.

Can AI agents use crawlie?

Yes. crawlie ships an MCP server and Claude Skills, so an agent can crawl a site, read structured findings, and explain fixes — something Screaming Frog can't do.

Does crawlie have a free URL limit?

No. crawlie crawls any number of URLs for free. Screaming Frog's free tier stops at 500 URLs per crawl.