Use case
One request.
The whole site, streaming.
POST /crawl turns a site into a stream: one request, NDJSON back, one complete JSON object per page flushed the moment it lands. No pagination loops, no job polling, no giant response buffering. Server memory stays constant whether the crawl is 50 pages or 50,000.
Every page gets the full ladder
The local crawl engine runs a BFS where each discovered URL is fetched with the same multi-vendor escalation as a /scrape call. A crawl doesn't die because page 400 sits behind a Cloudflare wall. That one page climbs the ladder while the stream keeps flowing.
Transformations per page, mid-stream
Pass formats: ["markdown"] (or text, html, screenshot) and each line arrives already transformed. Per-page errors arrive as in-band NDJSON error lines and the crawl continues; cancelling is just dropping the connection.
Cost you can watch in real time
Billing is per page, and every emitted line carries its own credits_charged, so your client can compute running cost mid-stream and stop the moment a budget is hit.
Start with a curl.
curl -sN "$GOTTEM_BASE_URL/crawl" \
-H "Authorization: Bearer $GOTTEM_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"limit": 50,
"depth": 2,
"formats": ["markdown"],
"return_links": true
}'Questions.
- How is /crawl billed?
- Per page, at the same route pricing as /scrape. Each NDJSON line includes its credits_charged, so running cost is visible mid-stream.
- What engines can a crawl use?
- auto (default), spider_cloud (a single round-trip to Spider's native crawler), or local (gottem-side BFS where every URL gets the full vendor ladder).