Describe the standard
Use the visual builder for titles, schema, phrases or page metrics.
Reach / Rules▊
Keep every page on-brand, on-spec and human.
Add the checks your team actually cares about, from product schema to tired AI phrases. Crawlie runs them on every crawl, so good intentions become part of the process.
Visual rule builder
Product pages need schema
Report as
Test against latest crawl
•ᴗ•Ready to testSee what this rule catches before saving it.
/products/monitorProductpass
/products/legacynoneflag
Preview it
Dry-run against your last crawl and see exactly what it would flag.
Enforce it everywhere
Every future crawl scores against your rules, with alerts on regressions.
Why it wins
Built different where it counts.
Build checks visually
Start with a template or build your own check. Choose the pages, the condition and how serious a miss should be.
Catch AI slop before Google does
Flag clichés, filler, repeated phrasing and the habits that make scaled content feel mass produced.
Preview before you commit
Test a rule against your last crawl, look at the matches, then switch it on when it feels right.
Content rules
An open format, not a black box
Rule packs are plain .crawlie files. They run the same way every time, work in CI, and can live in git beside the site.
- Deterministic checks with the same result every time
- Works in the cloud, the CLI, and CI
- Content score tracked crawl-over-crawl, with alerts on regressions
Products need schema 412 pass
No AI clichés 2 flagged
Brand terms spelled right all pass
Content score 8.2 → 7.9
Don’t start from blank
A rule library for the standards teams actually argue about.
Pick a useful starting point, change the scope, then test it against your latest crawl before it goes live.
Browse the built-in checks →Brand
Brand name in every titleKeep search snippets consistent
Voice
Ban tired AI phrasesFlag clichés and filler automatically
Trust
Authors on advice pagesRequire a named author on /guides/*
Schema
Product schema on product pagesCheck every /products/* route
7.0content score
+3
AI clichés“unlock the power of”
+2
Rule of three“fast, reliable and scalable”
+2
Filler heavyfiller ratio above 0.04
No black box at runtime
Same words in. Same result out.
Rules are deterministic. Every point traces back to the rule and the exact evidence that fired it, so writers can fix the sentence instead of debating a mystery score.
Phrase, regex and metric rulesSentence variance and lexical diversityRepeated n-grams and filler ratiosStructured parse errors with line and column
Guide: custom rules →One open format
Write it once. Run it everywhere.
01Visual builderFor the whole team
→02Cloud crawlsOn every scheduled run
→03CLI + CIBefore the page ships
→04AlertsWhen the standard slips
The `.crawlie` file can live in git beside the site. Your cloud workspace, local crawler and CI job all read the same rulebook.
Agent-native
Your agents get the same superpowers.
Everything on this page is a tool call away. The hosted MCP server exposes 18 tools — crawls, reports, fixes, monitoring — and the REST API speaks the same JSON. One key, no scraping, no glue code.
- Hosted MCP endpoint for Claude, Cursor and any MCP client
- REST API with the same capabilities, self-describing
- Open-source CLI + local MCP if you'd rather stay on your machine
> flag AI clichés on every crawl
list_rules → 148 built-in checks
explain_issue slop-cliche → why + fix
+ brand-voice.crawlie committed to git
crawl_site → scores your rules too
deterministic · same words, same score
What can a custom Crawlie rule check?
Phrases, regular expressions, page metrics and required structured data, scoped to the pages you choose — like requiring Product schema on /products/* or banning "in today's fast-paced world" everywhere. Each rule gets a severity: notice, warning or error.
What is a .crawlie file?
A plain-text rule pack. It can live in git beside your site, and the same file runs in your cloud workspace, the open-source CLI and CI — one rulebook, enforced everywhere.
Can Crawlie detect AI-generated filler content?
Yes. Built-in content rules flag tired AI clichés, heavy filler ratios, repeated n-grams and low lexical diversity — the habits that make scaled content feel mass-produced. Every point traces to the rule and the exact evidence that fired it.
Are rule results deterministic?
Completely. Rules are not an LLM judgment call — the same words in produce the same result out, every crawl, in every environment. That makes them safe to gate CI on.
Can I test a rule before enforcing it?
Yes — dry-run any rule against your latest crawl and see exactly which pages it would flag before switching it on. After that, every future crawl scores against it, with alerts on regressions.
Point Crawlie at your site.
Free to start on the Hobby plan — 5,000 crawl credits a month, no card required.