Crawlbase · Rate Limits

Crawlbase Rate Limits

Crawlbase governs throughput primarily by concurrent request threads and by your plan's monthly credit / request allowance rather than a single global per-minute cap. The default documented ceiling on the Crawling API is roughly 20 requests per second (the same default applies to the proxy interface at smartproxy.crawlbase.com); higher thread counts are unlocked by higher-tier subscriptions. Only requests that return pc_status 200 count against a pay-as-you-go quota - failed requests (timeouts, blocks, target 5xx) are free. Scroll-heavy JavaScript requests are metered by processing time (first 8 seconds = 1 request, each additional 5 seconds = 1 more request).

Crawlbase Rate Limits is the machine-readable rate-limit profile for Crawlbase on the APIs.io network, conforming to the API Commons Rate Limits specification.

It captures 4 rate-limit definitions, measuring requests, threads, and seconds.

The profile also includes 3 backoff/retry policies defined and response codes documented for throttled.

Tagged areas include Web Scraping, Web Crawling, Proxy, Rate Limiting, and Quotas.

4 Limits Throttle: 429
Web ScrapingWeb CrawlingProxyRate LimitingQuotas

Limits

Crawling API Request Rate account
requests
~20 requests/second (default)
Documented default ceiling; the proxy interface (smartproxy.crawlbase.com) shares the same default.
Concurrent Threads account
threads
per plan
Higher subscription tiers unlock more concurrent request threads.
Monthly Credit / Request Allowance account
requests
per plan
Pay-as-you-go and subscription tiers meter monthly successful requests / credits. Only pc_status 200 responses are charged.
Scroll Processing Metering request
seconds
1 request per first 8s, +1 per additional 5s
JavaScript scroll requests are billed by server processing time rather than a flat per-request cost.

Policies

Charge on Success Only
Only requests that return pc_status 200 count against quota; failed requests (timeouts, blocks, target 5xx) are free.
Backoff Strategy
Clients should implement exponential backoff with jitter and honor Retry-After on 429 responses.
Async Queueing
The Crawling API async=true mode queues requests and returns a request id, smoothing bursts and delivering results to a callback webhook.

Sources