Use case
Clean markdown in.
Grounded answers out.
A RAG pipeline is only as good as what it ingests. gottem fetches pages through whichever of its 14+ vendors can actually deliver them, runs the result through one transformation pipeline, and hands you consistent markdown. One parser, one response shape, regardless of which scraper did the work.
Markdown as a first-class output
Ask for formats: ["markdown"] and every page comes back LLM-ready. When a markdown-native vendor like Firecrawl serves the page you get its extraction; when another vendor serves it, gottem's transformation pipeline produces the same format server-side. Your chunking code never branches on vendor.
Ingestion that gets cheaper as it runs
Save State, gottem's per-domain route memory, remembers which route worked for each domain and goes straight there next time. Long-running ingestion jobs stop paying the exploration cost after the first pass over a domain.
Quality-gated content
Hit Detection, the content-quality gate, catches responses that return 200 but are really challenge pages or empty shells, so half-rendered junk never reaches your vector store.
Start with a curl.
curl -sN "$GOTTEM_BASE_URL/crawl" \
-H "Authorization: Bearer $GOTTEM_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://docs.example.com",
"limit": 200,
"depth": 3,
"formats": ["markdown"]
}'Questions.
- Do I get the same markdown regardless of vendor?
- Yes. One transformation pipeline runs server-side for every route, so the output format is consistent whether the page came through Firecrawl, Spider, or a heavier browser rung.
- How do I keep ingestion costs predictable?
- Pages start at $0.001, the ladder always tries the cheapest viable route first, and every NDJSON line carries its own credits_charged so you can compute running cost mid-stream.