# gottem

> One API for web scraping. Send a URL, get clean content back. gottem picks
> the fastest, most efficient way to fetch each page across major scraping
> providers. It escalates automatically when a site fights back. You pay
> only for requests that succeed.

gottem is the hosted scraping API at https://gottem.dev. It is built on the
open-source `gottem` engine (https://github.com/spider-rs/gottem) and unifies
the major scraping providers behind a single request shape, a single key, and
a single bill.

## Why gottem

- **Reliably gets the data.** One request tries the fastest, lowest-cost viable
  path first. It escalates to proxies, headless browsers, unblockers, and
  captcha solving when a site blocks it.
- **Major providers, one API.** Firecrawl, ZenRows, Zyte, Brightdata,
  Apify, Oxylabs, Browserbase, Spider, and more use one schema.
  Switch providers without rewriting anything.
- **Tuned per request.** Choose lowest cost, lowest latency, or reliability.
  Race providers in parallel or hedge with staggered backups when speed matters.
- **Learns every site.** gottem remembers what works for each domain and goes
  straight there next time. Routing gets faster and more efficient as you use it.
- **Bring your own keys (BYOK).** Use keys from vendors you already pay.
  gottem routes through them for a flat infrastructure fee.
- **Safe by default.** Known-malicious hosts (phishing, malware, scams) are
  filtered before a request is made.

## API

Base URL: `https://api.gottem.dev`. Authenticate with an API key:
`Authorization: Bearer gtm_…` (create one in the dashboard). Every endpoint
except `/healthz` requires a key.

- `POST /scrape`: `{ url, mode?, require_js?, ... }` → the page content plus
  metadata (which route served it, tier, credits charged, elapsed, content
  quality).
- `POST /crawl`: `{ url, limit?, depth?, engine?, formats?, return_links?, ... }`
  → streaming NDJSON, **one JSON object per line**, flushed as each page
  lands. Server memory stays constant regardless of crawl size. Engines:
  `auto` (default), `spider_cloud` (vendor's native `/crawl`, single
  round-trip), `local` (gottem-side BFS over the scrape ladder — every URL
  re-uses the full provider escalation, link discovery uses spider's
  primitives on already-fetched bytes). The `formats` field runs the same
  transformation pipeline `/scrape` uses, per page, so any vendor's output
  becomes markdown / html / text / screenshot on demand. Per-page billing.
- `POST /probe`: test which routes can reach a URL.
- `POST /v1/compare`: run a URL through several providers at once and compare
  what each returns. Built for quality validation: every provider's
  content-quality verdict, cost, and a SHA-256 of its content, side by side.
  Deterministic; identical results are merged into `variants`.
- `GET /routes`: the available routes.
- `GET /healthz`: service health (no auth; the only open endpoint).

`mode` is one of `ladder` (lowest-cost-first), `race` (parallel, fastest wins),
or `hedge` (primary plus staggered backups).

Account management (keys, credits, top-ups, auto-recharge) is done from the
dashboard, or via the control-plane API at `https://query.gottem.dev`.

## Pricing

Pay-as-you-go, credit based. **1 credit = $0.0001**, or 10,000 credits per US
dollar. Credits are debited only on a successful fetch. Top up any time. No
subscriptions, no monthly minimum, no expiry.

## Links

- Website: https://gottem.dev
- API docs / reference: https://gottem.dev/docs
- Pricing: https://gottem.dev/#pricing
- Sign in / dashboard: https://gottem.dev/signin
- Privacy: https://gottem.dev/privacy
- Terms: https://gottem.dev/terms
- API base (data plane): https://api.gottem.dev
- Control plane (auth, billing): https://query.gottem.dev
- Open-source engine: https://github.com/spider-rs/gottem