Galon Cloud
Galon API v1

/search/v2/crawl

Base URL https://api.galon.ai/v1 · export GALON_API_BASE="https://api.galon.ai/v1"

POST /search/v2/crawl

Start a crawl job (read a site)

Starts a background reading of a site: breadth-first from the start address (and its sitemaps), within the start path unless crawlEntireDomain, at most `limit` pages (default 10, capped at 100 and said in `warning`), `maxDepth` hops (default 2, at most 5), 15 minutes, 2 running jobs per project. `Idempotency-Key` (optional): the same key with the same request returns the same job (header Idempotent-Replayed: true); a different request with the same key is 409. Each page read is counted. Results are kept 24 hours.

API key (Bearer), on the public API endpoint.

Parameters

NameInTypeDescription
Idempotency-Keyheaderstring · required

Request body

application/json · schema CrawlStartRequest

FieldTypeDescription
urlstring · required
limitintegerValues above 100 are capped (warning limit_capped).
maxDepthinteger
maxDiscoveryDepthintegerAlias of maxDepth (not both).
includePathsarray<string>Regular expressions on the path; nested repetition refused (pattern_unsafe).
excludePathsarray<string>
crawlEntireDomainboolean
allowBackwardLinksbooleanAlias of crawlEntireDomain.
allowSubdomainsboolean
allowExternalLinksbooleanRefused unless false (option_unsupported).
ignoreSitemapboolean
sitemap"include" | "skip" | "only"
ignoreQueryParametersboolean
delaynumberSeconds between pages; never below 1.
maxConcurrencyintegerAccepted without effect: one page at a time per job.
scrapeOptionsCrawlScrapeOptions
webhookanyRefused unless empty (option_unsupported).
promptstringRefused unless empty (option_unsupported).

Responses

StatusDescriptionBody
200Job created (or replayed)CrawlStartResponse
400body_invalid, field_unknown, field_invalid, url_invalid, url_blocked, option_unsupported (webhook, prompt, allowExternalLinks, regexOnFullURL), pattern_unsafe, javascript_rendering_unavailableCrawlError
401api_key_missing, api_key_invalidCrawlError
402insufficient_balance (reservation of limit × page price, only when a price grid is set)CrawlError
403scope_missing (search:query)CrawlError
404not_foundCrawlError
409idempotency_conflictCrawlError
429crawl_limit_reached, daily_limit_reached, quota_exceeded, tenant_quota_exceededCrawlError
503search_unavailableCrawlError

Examples

curl

curl -X POST "$GALON_API_BASE/search/v2/crawl" \
  -H "Authorization: Bearer $GALON_API_KEY" \
  -H "Idempotency-Key: <unique-key>" \
  -H "Content-Type: application/json" \
  -d '{"url": "string"}'

Node.js (SDK)

import { Galon } from "galon";

const galon = new Galon(process.env.GALON_API_KEY);
const result = await galon.search.startCrawl({"url": "string"}, { idempotencyKey: "<unique-key>" });

Python (SDK)

from galon import Galon

galon = Galon()  # reads GALON_API_KEY
result = galon.search.start_crawl({"url": "string"}, idempotency_key="<unique-key>")