# Galon Search

> Read web pages and sites for your AI: scrape a page, map a site's links, crawl a site to Markdown.

## Before you start

- A Galon account, a project and a key with `search:query`.

## Quickstart

Every example reads `GALON_API_BASE` (`https://api.galon.ai/v1`) and `GALON_API_KEY` from the environment and prints the HTTP status and the JSON body.

**1. Set your key.** Export the base URL and your key (console: Keys, scope `search:query`).

**curl**

```bash
export GALON_API_BASE="https://api.galon.ai/v1"
export GALON_API_KEY="gln_..."
```

**2. Scrape a page.** `formats`: `markdown` (default), `html`, `rawHtml`, `links`. JavaScript rendering is not offered (`javascript_rendering_unavailable`); `timeout` above 60000 ms is capped.

**curl**

```bash
curl -sS -X POST "$GALON_API_BASE/search/v2/scrape" \
  -H "Authorization: Bearer $GALON_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}' \
  -w '\n%{http_code}\n'
```

**Python 3 (standard library)**

```python
import json, os, urllib.error, urllib.request

req = urllib.request.Request(
    os.environ["GALON_API_BASE"] + "/search/v2/scrape",
    data=json.dumps({"url": "https://example.com", "formats": ["markdown"]}).encode(),
    headers={"Authorization": "Bearer " + os.environ["GALON_API_KEY"],
             "Content-Type": "application/json"},
    method="POST",
)
try:
    with urllib.request.urlopen(req) as res:
        print(res.status, json.load(res))
except urllib.error.HTTPError as err:  # the error body carries detail.code and detail.next_action
    print(err.code, json.load(err))
```

**Node.js 18+ (fetch)**

```js
const res = await fetch(`${process.env.GALON_API_BASE}/search/v2/scrape`, {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.GALON_API_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({url: "https://example.com", formats: ["markdown"]}),
});
console.log(res.status, JSON.stringify(await res.json()));
```

**3. Map a site.** The addresses of a site's pages (sitemaps and start page), without reading them; `search` puts the addresses that contain a text first.

**curl**

```bash
curl -sS -X POST "$GALON_API_BASE/search/v2/map" \
  -H "Authorization: Bearer $GALON_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "search": "pricing", "limit": 100}' \
  -w '\n%{http_code}\n'
```

**Python 3 (standard library)**

```python
import json, os, urllib.error, urllib.request

req = urllib.request.Request(
    os.environ["GALON_API_BASE"] + "/search/v2/map",
    data=json.dumps({"url": "https://example.com", "search": "pricing", "limit": 100}).encode(),
    headers={"Authorization": "Bearer " + os.environ["GALON_API_KEY"],
             "Content-Type": "application/json"},
    method="POST",
)
try:
    with urllib.request.urlopen(req) as res:
        print(res.status, json.load(res))
except urllib.error.HTTPError as err:  # the error body carries detail.code and detail.next_action
    print(err.code, json.load(err))
```

**Node.js 18+ (fetch)**

```js
const res = await fetch(`${process.env.GALON_API_BASE}/search/v2/map`, {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.GALON_API_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({url: "https://example.com", search: "pricing", limit: 100}),
});
console.log(res.status, JSON.stringify(await res.json()));
```

**4. Crawl a site.** A background job: `limit` pages (default 10, at most 100), `maxDepth` hops (at most 5), 15 minutes, 2 jobs at a time per project. Send an `Idempotency-Key`: the same key returns the same job, never a second one.

**curl**

```bash
curl -sS -X POST "$GALON_API_BASE/search/v2/crawl" \
  -H "Authorization: Bearer $GALON_API_KEY" \
  -H "Idempotency-Key: docs-2026-11-14" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/docs", "limit": 25, "maxDepth": 2, "excludePaths": ["/docs/archive/.*"]}' \
  -w '\n%{http_code}\n'
```

**Python 3 (standard library)**

```python
import json, os, urllib.error, urllib.request

req = urllib.request.Request(
    os.environ["GALON_API_BASE"] + "/search/v2/crawl",
    data=json.dumps({"url": "https://example.com/docs", "limit": 25, "maxDepth": 2, "excludePaths": ["/docs/archive/.*"]}).encode(),
    headers={"Authorization": "Bearer " + os.environ["GALON_API_KEY"],
             "Idempotency-Key": "docs-2026-11-14",
             "Content-Type": "application/json"},
    method="POST",
)
try:
    with urllib.request.urlopen(req) as res:
        print(res.status, json.load(res))
except urllib.error.HTTPError as err:  # the error body carries detail.code and detail.next_action
    print(err.code, json.load(err))
```

**Node.js 18+ (fetch)**

```js
const res = await fetch(`${process.env.GALON_API_BASE}/search/v2/crawl`, {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.GALON_API_KEY}`, "Idempotency-Key": "docs-2026-11-14", "Content-Type": "application/json" },
  body: JSON.stringify({url: "https://example.com/docs", limit: 25, maxDepth: 2, excludePaths: ["/docs/archive/.*"]}),
});
console.log(res.status, JSON.stringify(await res.json()));
```

**5. Read the crawl's pages.** `status` is `scraping`, `completed`, `failed` or `cancelled`; 20 pages per answer, follow `next` while present. Results are kept 24 hours. `DELETE /search/v2/crawl/{id}` stops a running job.

**curl**

```bash
curl -sS -X GET "$GALON_API_BASE/search/v2/crawl/{id}" \
  -H "Authorization: Bearer $GALON_API_KEY" \
  -w '\n%{http_code}\n'
```

**Python 3 (standard library)**

```python
import json, os, urllib.error, urllib.request

req = urllib.request.Request(
    os.environ["GALON_API_BASE"] + "/search/v2/crawl/{id}",
    headers={"Authorization": "Bearer " + os.environ["GALON_API_KEY"]},
    method="GET",
)
try:
    with urllib.request.urlopen(req) as res:
        print(res.status, json.load(res))
except urllib.error.HTTPError as err:  # the error body carries detail.code and detail.next_action
    print(err.code, json.load(err))
```

**Node.js 18+ (fetch)**

```js
const res = await fetch(`${process.env.GALON_API_BASE}/search/v2/crawl/{id}`, {
  method: "GET",
  headers: { Authorization: `Bearer ${process.env.GALON_API_KEY}` },
});
console.log(res.status, JSON.stringify(await res.json()));
```

## Concepts

**Scrape, map, crawl.** `POST /v1/search/v2/scrape` reads one page; `/map` lists a site's links; `/crawl` starts a job (`GET /v1/search/v2/crawl/{id}` pages through its results, `DELETE` cancels it). A SDK written for the market's v2 API migrates by changing its base URL and key; an option Galon does not serve is refused by name, never ignored.

**Polite by design.** One request per second per host, robots.txt obeyed (unreachable robots.txt: the site is not read), user agent `GalonBot/1.0 (+https://galon.ai/bot)`, public pages only: private, local and reserved networks are refused.

**Try it in the console.** Services → Search → Read websites: one address, one button; the page's Markdown, the links, a site reading, and this first call with your project's address.

**Counting.** One page for scrape and crawl, one call for map; a refused page is not counted. Monthly pages per project and per account, daily requests per project: 429 by name when reached.

**Retention.** Crawl results are kept 24 hours; scrape and map keep no content.

## API reference

These routes are served by the code but are not in the OpenAPI contract yet:

| Method | Path | Auth | What it does |
|---|---|---|---|
| POST | `/v1/search/v2/scrape` | key `search:query` | Read one page |
| POST | `/v1/search/v2/map` | key `search:query` | List a site's links |
| POST | `/v1/search/v2/crawl` | key `search:query` | Start a crawl |
| GET | `/v1/search/v2/crawl/{id}` | key `search:query` | Read a crawl's results |
| GET | `/v1/search/v2/crawl/{id}/errors` | key `search:query` | Pages a crawl could not read |
| DELETE | `/v1/search/v2/crawl/{id}` | key `search:query` | Cancel a crawl |

## Errors

Every error is JSON: `{"detail": {"code", "message", "next_action"}}`. `next_action.code` is stable, `next_action.hint` says what to do next, and `console_url` is added when a person must act. The codes below are those this service returns.

| Status | Code | When | next_action |
|---|---|---|---|
| 403 | `scope_missing` | The key lacks `search:query`. | `add_missing_scope` |
| 400 | `field_unknown` | A field the API does not accept. | `fix_request` |
| 400 | `url_invalid` | Not an absolute http(s) URL on a standard port. | `fix_request` |
| 400 | `url_blocked` | A private, local or reserved address (also after a redirect). | `fix_request` |
| 400 | `robots_disallowed` | The site's robots.txt forbids the page. | `fix_request` |
| 400 | `format_unsupported` | A format that is not served (screenshot, json, …). | `fix_request` |
| 400 | `javascript_rendering_unavailable` | `waitFor` or `actions` (JavaScript rendering). | `fix_request` |
| 400 | `option_unsupported` | An option that is not served (headers, location, webhook, prompt…). | `fix_request` |
| 429 | `rate_limited` | Too many requests to one site; `Retry-After` says when. | `retry_after_delay` |
| 429 | `key_rate_limited` | Too many POST requests with one key in a minute (default 60); `Retry-After`. | `retry_after_delay` |
| 429 | `daily_limit_reached` | The daily request limit is reached. | `raise_limit_or_plan` |
| 429 | `quota_exceeded` | The monthly page quota is reached. | `raise_limit_or_plan` |
| 429 | `crawl_limit_reached` | Two crawl jobs already run for this project. | `raise_limit_or_plan` |
| 409 | `idempotency_conflict` | The same `Idempotency-Key` with a different request. | `resolve_conflict` |
| 404 | `crawl_not_found` | No crawl with this id. | `check_resource_id` |
| 404 | `crawl_expired` | The crawl's results are older than 24 hours. | `check_resource_id` |

| next_action | What to do |
|---|---|
| `add_missing_scope` | This key does not have the scope this call needs. Create a key with that scope in the console, then retry the same call with the new key. |
| `fix_request` | The request is invalid. Fix it as 'message' and 'field' say, using the OpenAPI contract, then send it again. Do not retry it unchanged. |
| `retry_after_delay` | Too many requests. Wait the Retry-After delay (seconds), then retry with exponential backoff and the same Idempotency-Key. |
| `raise_limit_or_plan` | A plan quota or a spending limit is reached. Raise the limit or change the plan in the console, or wait for the next period, then retry. |
| `resolve_conflict` | The request conflicts with the resource state or with an earlier request under the same Idempotency-Key. Read the resource first; for a new intent, use a new Idempotency-Key. |
| `check_resource_id` | Nothing with this id is visible to this key. List the resources to get a valid id, and check that the key belongs to the right project and environment (test or live). |

## Limits, free tier and pricing

Read from the platform code each time this page is served.

| Limit | Value |
|---|---|
| Crawl pages (default limit) | **10** |
| Crawl pages (maximum per job) | **100** |
| Crawl depth (maximum) | **5** |
| Map links (maximum) | **1,000** |
| Page size (bytes) | **5 MiB** |
| Results kept (hours) | **24** |
| Concurrent crawl jobs per project | **2** |
| Pages per site per day (default) | **500** |
| Site reading requests per minute per key (default) | **60** |
| Page timeout (milliseconds, maximum) | **60,000** |
| Indexes per project (default) | **10** |
| Documents per index (default) | **100,000** |
| Documents per account (default) | **500,000** |
| Document size | **64 KiB** |
| Documents per batch | **1,000** |
| Queries per second per key (default) | **20** |

Usage prices, in US dollars, read from the price grid served by the platform each time this page is served. Paid from the prepaid balance.

| Item | Price | Unit |
|---|---|---|
| `page` | 0.741 USD | per 1,000 documents |
| `call` | 0.741 USD | per 1,000 requests |

Billing units: one page for scrape and crawl, one call for map, paid from the prepaid balance.

## FAQ

**Is this a web search API?**

No: it reads pages and sites you name. Search over an index is not on this page.

**Can it render JavaScript?**

No: the page is read as served. `waitFor` and `actions` are refused by name.

**Can I try it without code?**

Yes: in the console, Services → Search → Read websites.

**How big can a crawl be?**

100 pages and 5 levels per job, 15 minutes; the request's `limit` above 100 is capped and the answer says so (`warning`).

## More

- [Authentication](/cloud/docs/platform/authentication)
- [Projects: live and test](/cloud/docs/platform/environments)
- [Prepaid balance](/cloud/docs/platform/balance)
- [Quotas and rate limits](/cloud/docs/platform/quotas)
- [Terms of Service](/legal/terms-of-service)
- [Acceptable Use Policy](/legal/acceptable-use-policy)
- [Privacy Policy](/legal/privacy-policy)
- [Refunds and prepaid balance](/legal/prepaid-balance-and-refund-policy)
- [Legal Notice](/legal/legal-notice)
- [All legal documents](/legal)
