POST /search/v2/crawl
Start a crawl job (read a site)
Starts a background reading of a site: breadth-first from the start address (and its sitemaps), within the start path unless crawlEntireDomain, at most `limit` pages (default 10, capped at 100 and said in `warning`), `maxDepth` hops (default 2, at most 5), 15 minutes, 2 running jobs per project. `Idempotency-Key` (optional): the same key with the same request returns the same job (header Idempotent-Replayed: true); a different request with the same key is 409. Each page read is counted. Results are kept 24 hours.
API key (Bearer), on the public API endpoint.
Parameters
| Name | In | Type | Description |
|---|---|---|---|
Idempotency-Key | header | string · required |
Request body
application/json · schema CrawlStartRequest
| Field | Type | Description |
|---|---|---|
url | string · required | |
limit | integer | Values above 100 are capped (warning limit_capped). |
maxDepth | integer | |
maxDiscoveryDepth | integer | Alias of maxDepth (not both). |
includePaths | array<string> | Regular expressions on the path; nested repetition refused (pattern_unsafe). |
excludePaths | array<string> | |
crawlEntireDomain | boolean | |
allowBackwardLinks | boolean | Alias of crawlEntireDomain. |
allowSubdomains | boolean | |
allowExternalLinks | boolean | Refused unless false (option_unsupported). |
ignoreSitemap | boolean | |
sitemap | "include" | "skip" | "only" | |
ignoreQueryParameters | boolean | |
delay | number | Seconds between pages; never below 1. |
maxConcurrency | integer | Accepted without effect: one page at a time per job. |
scrapeOptions | CrawlScrapeOptions | |
webhook | any | Refused unless empty (option_unsupported). |
prompt | string | Refused unless empty (option_unsupported). |
Responses
| Status | Description | Body |
|---|---|---|
200 | Job created (or replayed) | CrawlStartResponse |
400 | body_invalid, field_unknown, field_invalid, url_invalid, url_blocked, option_unsupported (webhook, prompt, allowExternalLinks, regexOnFullURL), pattern_unsafe, javascript_rendering_unavailable | CrawlError |
401 | api_key_missing, api_key_invalid | CrawlError |
402 | insufficient_balance (reservation of limit × page price, only when a price grid is set) | CrawlError |
403 | scope_missing (search:query) | CrawlError |
404 | not_found | CrawlError |
409 | idempotency_conflict | CrawlError |
429 | crawl_limit_reached, daily_limit_reached, quota_exceeded, tenant_quota_exceeded | CrawlError |
503 | search_unavailable | CrawlError |
Examples
curl
curl -X POST "$GALON_API_BASE/search/v2/crawl" \
-H "Authorization: Bearer $GALON_API_KEY" \
-H "Idempotency-Key: <unique-key>" \
-H "Content-Type: application/json" \
-d '{"url": "string"}'Node.js (SDK)
import { Galon } from "galon";
const galon = new Galon(process.env.GALON_API_KEY);
const result = await galon.search.startCrawl({"url": "string"}, { idempotencyKey: "<unique-key>" });Python (SDK)
from galon import Galon
galon = Galon() # reads GALON_API_KEY
result = galon.search.start_crawl({"url": "string"}, idempotency_key="<unique-key>")