Site search

Search built from your crawls

Crawlie already crawls every page of your site, so it can also make those pages searchable. Crawlie Search turns any crawl into a hosted full-text index (BM25 ranking, typo tolerance, phrase queries) with a ready-made search dialog you can embed anywhere — no separate search service to run, no re-crawling, no content pipeline to maintain.

Indexes are their own thing, composed from one or more sources:

• A monitored site’s latest crawl — the index rebuilds automatically every time that project crawls, so it’s always as fresh as your monitoring schedule.
• A specific report — pin an index to one crawl when you want it frozen.

Each source can be narrowed to route prefixes. Only index /help/, exclude /help/archive/, or combine several sites into one corpus — whatever shape the search surface needs.

Set up an index

1. Open Search in the dashboard sidebar and click New index.
2. Name it (e.g. Docs search), then add sources: pick a monitored site or a specific report, and optionally limit it to routes like /help/ or /docs/.
3. Click Create index. Crawlie indexes the matching pages in seconds and shows a test search box so you can try real queries immediately.

Rebuild any time from the index’s Rebuild button; indexes with a monitored site as a source rebuild themselves after every crawl of that site.

Install it on your site

Every index has its own embed snippet — one line of HTML:

<script src="https://crawlie.app/search.js" data-project="YOUR_INDEX_TOKEN"></script>

That gives you a styled, keyboard-friendly search dialog (⌘K and / work out of the box, dark mode included). Add data-launcher="off" to hide the floating button and open it from your own UI instead — any element with a data-crawlie-search attribute, or window.crawlieSearch.open().

Prefer to build your own UI? Query the index directly — it’s a plain GET, no auth needed:

GET https://crawlie.app/pub/search/YOUR_INDEX_TOKEN?q=your+query

Use cases

• Docs & help-center search — index just your /docs/ or /help/ routes so results are answers, never landing pages. This help center runs exactly that: press ⌘K here and you’re using a Crawlie Search index of /help/ routes.
• Whole-site search — a search box for your marketing site with zero build changes. The search on crawlie.co is our own crawl.
• Multi-site search — merge several properties (main site + docs + blog on different domains) into one index with one search box.
• Search quality checks — use the dashboard’s test box to see what a query actually surfaces on your site, and which pages are missing from the corpus.

Pricing

Crawlie Search is free while in beta, on every plan — including Hobby. Indexes are built from crawls you’ve already paid for (or free-plan crawls), so there’s no extra metering. Pricing may change when Search leaves beta; we’ll announce it in the changelog well ahead of time.

Good to know

• An index reflects the latest completed crawl of its sources — pages published since then appear after the next crawl (a good reason to schedule one).
• The public endpoint and widget only expose what the crawl saw: titles, URLs, and text excerpts from public pages. Pages that are noindex or canonicalised elsewhere are left out.
• Deleting an index revokes its public token immediately; your crawls and reports are untouched.

Crawlie Search is new and we’re shaping it in the open — tell us what you’re building with it (or what’s missing) via Share feedback in the dashboard sidebar.