Retrieve, list, and delete pages that Crawlbase persisted when a crawl was sent with `&store=true`. Fetch a stored page by `rid` or `url`, pull up to 100 at once via bulk endpoints, page through all request IDs, and read the total stored count. Pages are retained 14 days by default; storage writes and reads are not separately billed.
openapi: 3.0.3
info:
title: Crawlbase API
description: >-
Crawlbase (formerly ProxyCrawl) is a web crawling and scraping platform.
A single REST host, https://api.crawlbase.com, exposes several products,
all authenticated with a `token` query parameter: the Crawling API (fetch
any URL through a rotating proxy network, optionally rendered in headless
Chrome), the Scraper API (ready-made structured extractors for popular
sites), the Cloud Storage API (retrieve/list/delete previously stored
crawls), the Screenshots API (rendered page captures), and the Leads API
(domain email discovery).
Grounding note: paths, methods, and the token query-param auth model are
confirmed from Crawlbase's public documentation. Request query parameters
are modeled from the documented parameter reference. Response bodies are
largely raw upstream content (HTML, Markdown, JSON, or images) or, for the
Scraper API, provider-specific JSON whose exact fields vary per scraper and
are therefore modeled loosely rather than enumerated. Treat response schemas
as illustrative.
version: '1.0'
contact:
name: Crawlbase
url: https://crawlbase.com
servers:
- url: https://api.crawlbase.com
description: Crawlbase API host (all products share this host)
security:
- tokenAuth: []
tags:
- name: Crawling API
description: Fetch any URL through the rotating proxy network, optionally rendered.
- name: Scraper API
description: Ready-made structured-data extractors for supported sites (legacy).
- name: Storage API
description: Retrieve, list, and delete previously stored crawls.
- name: Screenshots API
description: Rendered page screenshots in headless Chrome (legacy).
- name: Leads API
description: Publicly visible email discovery for a domain (legacy).
paths:
/:
get:
operationId: crawl
tags:
- Crawling API
summary: Crawl a URL
description: >-
Fetches the target `url` through Crawlbase's proxy network and returns
the page. Use the Normal (TCP) token for static HTML/JSON, or the
JavaScript token to render the page in headless Chrome before returning
it. Crawlbase status is surfaced in the `pc_status` response header;
requests that do not return `pc_status: 200` are not charged.
parameters:
- $ref: '#/components/parameters/Url'
- $ref: '#/components/parameters/Format'
- $ref: '#/components/parameters/Country'
- $ref: '#/components/parameters/Device'
- $ref: '#/components/parameters/UserAgent'
- $ref: '#/components/parameters/Scraper'
- name: autoparse
in: query
required: false
description: Auto-detect the page type and return parsed JSON.
schema:
type: boolean
- name: page_wait
in: query
required: false
description: Milliseconds to wait after page load before capture (JavaScript token only).
schema:
type: integer
- name: ajax_wait
in: query
required: false
description: Wait for AJAX / network idle before capture (JavaScript token only).
schema:
type: boolean
- name: css_click_selector
in: query
required: false
description: CSS selector to click before capture (JavaScript token only).
schema:
type: string
- name: scroll
in: query
required: false
description: Scroll toward the bottom of the page before capture (JavaScript token only).
schema:
type: boolean
- name: scroll_interval
in: query
required: false
description: Maximum seconds to scroll (default 10, max 60).
schema:
type: integer
- name: screenshot
in: query
required: false
description: Capture a JPEG screenshot of the rendered page (JavaScript token only).
schema:
type: boolean
- name: get_headers
in: query
required: false
description: Return the target's response headers alongside the body.
schema:
type: boolean
- name: get_cookies
in: query
required: false
description: Return the target's Set-Cookie values.
schema:
type: boolean
- name: cookies_session
in: query
required: false
description: Sticky session identifier to reuse cookies across requests.
schema:
type: string
- name: store
in: query
required: false
description: Persist the response in Crawlbase Cloud Storage.
schema:
type: boolean
- name: async
in: query
required: false
description: Queue the request and return immediately with a request id (rid).
schema:
type: boolean
- name: callback
in: query
required: false
description: Webhook URL that Crawlbase posts async results to.
schema:
type: string
format: uri
responses:
'200':
description: >-
The crawled page. Body is raw upstream content (HTML, Markdown, JSON,
or image) depending on `format`, `scraper`, and `screenshot`.
Crawlbase-specific metadata (pc_status, original_status, rid, url) is
returned in response headers, or inline when `format=json`.
content:
text/html:
schema:
type: string
application/json:
schema:
type: object
additionalProperties: true
'401':
$ref: '#/components/responses/Unauthorized'
post:
operationId: crawlPost
tags:
- Crawling API
summary: Crawl a URL (POST)
description: >-
Same as the GET crawl operation, but the request payload (for example,
POST form data to submit to the target) is sent in the request body.
The `token` and `url` remain query parameters.
parameters:
- $ref: '#/components/parameters/Url'
- $ref: '#/components/parameters/Format'
- $ref: '#/components/parameters/Country'
- $ref: '#/components/parameters/Device'
requestBody:
required: false
content:
application/x-www-form-urlencoded:
schema:
type: object
additionalProperties: true
responses:
'200':
description: The crawled page.
content:
text/html:
schema:
type: string
application/json:
schema:
type: object
additionalProperties: true
'401':
$ref: '#/components/responses/Unauthorized'
/scraper:
get:
operationId: scrape
tags:
- Scraper API
summary: Scrape a supported site to structured JSON
description: >-
Applies a named `scraper` to the target `url` and returns provider-specific
structured JSON (for example, amazon-product-details, google-serp,
linkedin-company). Documented by Crawlbase as a legacy endpoint; the
Crawling API's `&scraper=` parameter offers the same extractors.
parameters:
- $ref: '#/components/parameters/Url'
- name: scraper
in: query
required: true
description: Name of the scraper/extractor to apply (see the Crawlbase scraper catalog).
schema:
type: string
- $ref: '#/components/parameters/Country'
- name: javascript
in: query
required: false
description: Render the page in Chrome before scraping (requires the JavaScript token).
schema:
type: boolean
- name: premium
in: query
required: false
description: Route through the premium residential proxy network.
schema:
type: boolean
responses:
'200':
description: Structured JSON for the scraped page. Fields vary per scraper.
content:
application/json:
schema:
type: object
additionalProperties: true
'401':
$ref: '#/components/responses/Unauthorized'
/storage:
get:
operationId: getStoredPage
tags:
- Storage API
summary: Retrieve a stored page
description: >-
Returns a previously stored crawl by `rid` or by `url`. When queried by
`url`, the most recent stored version is returned. Exactly one of `rid`
or `url` is required.
parameters:
- name: rid
in: query
required: false
description: The request id of a stored crawl.
schema:
type: string
- name: url
in: query
required: false
description: The URL-encoded target URL whose most recent stored version to return.
schema:
type: string
- $ref: '#/components/parameters/Format'
responses:
'200':
description: The stored page.
content:
text/html:
schema:
type: string
application/json:
schema:
type: object
additionalProperties: true
'401':
$ref: '#/components/responses/Unauthorized'
'404':
$ref: '#/components/responses/NotFound'
delete:
operationId: deleteStoredPage
tags:
- Storage API
summary: Delete a stored page
description: Deletes a single stored crawl identified by its `rid`.
parameters:
- name: rid
in: query
required: true
description: The request id of the stored crawl to delete.
schema:
type: string
responses:
'200':
description: Deletion confirmation.
content:
application/json:
schema:
type: object
additionalProperties: true
'401':
$ref: '#/components/responses/Unauthorized'
'404':
$ref: '#/components/responses/NotFound'
/storage/bulk:
post:
operationId: getStoredPagesBulk
tags:
- Storage API
summary: Bulk retrieve stored pages
description: Fetches up to 100 stored crawls in a single request by a list of request ids.
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
rids:
type: array
items:
type: string
responses:
'200':
description: A batch of stored pages.
content:
application/json:
schema:
type: object
additionalProperties: true
'401':
$ref: '#/components/responses/Unauthorized'
/storage/bulk_delete:
post:
operationId: deleteStoredPagesBulk
tags:
- Storage API
summary: Bulk delete stored pages
description: Deletes multiple stored crawls in a single request by a list of request ids.
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
rids:
type: array
items:
type: string
responses:
'200':
description: Deletion confirmation.
content:
application/json:
schema:
type: object
additionalProperties: true
'401':
$ref: '#/components/responses/Unauthorized'
/storage/rids:
get:
operationId: listStoredRids
tags:
- Storage API
summary: List stored request ids
description: Lists stored request ids with scroll-based pagination.
parameters:
- name: limit
in: query
required: false
description: Maximum request ids to return per page (cap 10000).
schema:
type: integer
- name: scroll
in: query
required: false
description: Enable scroll pagination; the response returns a scroll_id to continue.
schema:
type: boolean
responses:
'200':
description: A page of request ids.
content:
application/json:
schema:
type: object
additionalProperties: true
'401':
$ref: '#/components/responses/Unauthorized'
/storage/total_count:
get:
operationId: getStoredTotalCount
tags:
- Storage API
summary: Total stored page count
description: Returns the total number of pages currently in Cloud Storage.
responses:
'200':
description: The total stored count.
content:
application/json:
schema:
type: object
properties:
totalCount:
type: integer
'401':
$ref: '#/components/responses/Unauthorized'
/screenshots:
get:
operationId: screenshot
tags:
- Screenshots API
summary: Capture a screenshot of a URL
description: >-
Renders the target `url` in headless Chrome and returns a screenshot.
Requires the JavaScript token. Documented by Crawlbase as a legacy
endpoint (closed to new sign-ups on 2024-11-01).
parameters:
- $ref: '#/components/parameters/Url'
- name: mode
in: query
required: false
description: viewport (visible area) or fullpage (entire scrollable page).
schema:
type: string
enum:
- viewport
- fullpage
default: viewport
- name: format
in: query
required: false
description: Image output format.
schema:
type: string
enum:
- png
- jpeg
default: png
- name: width
in: query
required: false
description: Viewport width in pixels.
schema:
type: integer
default: 1280
- name: height
in: query
required: false
description: Viewport height in pixels.
schema:
type: integer
default: 800
- $ref: '#/components/parameters/Device'
- name: store
in: query
required: false
description: Persist the capture in Cloud Storage.
schema:
type: boolean
responses:
'200':
description: The rendered screenshot image.
content:
image/png:
schema:
type: string
format: binary
image/jpeg:
schema:
type: string
format: binary
'401':
$ref: '#/components/responses/Unauthorized'
/leads:
get:
operationId: getLeads
tags:
- Leads API
summary: Discover emails for a domain
description: >-
Returns publicly visible email addresses associated with a bare `domain`,
with the source URLs where each was found. Billed at one credit per ten
emails. Documented by Crawlbase as closed to new sign-ups since
2024-10-01.
parameters:
- name: domain
in: query
required: true
description: Bare domain (no scheme or path), for example slack.com.
schema:
type: string
- name: limit
in: query
required: false
description: Maximum emails to return (default 10).
schema:
type: integer
default: 10
responses:
'200':
description: Discovered emails with source URLs and available metadata.
content:
application/json:
schema:
type: object
additionalProperties: true
'401':
$ref: '#/components/responses/Unauthorized'
components:
securitySchemes:
tokenAuth:
type: apiKey
in: query
name: token
description: >-
Crawlbase authentication token passed as the `token` query parameter.
Each account has two tokens: a Normal (TCP) token for static content and
a JavaScript token for headless-Chrome rendering.
parameters:
Url:
name: url
in: query
required: true
description: Fully URL-encoded target URL, including http:// or https://.
schema:
type: string
Format:
name: format
in: query
required: false
description: Response format - html (default), json, or md (Markdown).
schema:
type: string
enum:
- html
- json
- md
default: html
Country:
name: country
in: query
required: false
description: Two-letter ISO 3166 country code to geolocate the request (for example US, DE, GB).
schema:
type: string
Device:
name: device
in: query
required: false
description: Device profile - desktop (default), tablet, or mobile.
schema:
type: string
enum:
- desktop
- tablet
- mobile
default: desktop
UserAgent:
name: user_agent
in: query
required: false
description: Override the User-Agent header sent to the target.
schema:
type: string
Scraper:
name: scraper
in: query
required: false
description: Name of a built-in extractor to apply to the crawled page (returns parsed JSON).
schema:
type: string
responses:
Unauthorized:
description: Missing or invalid token.
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
NotFound:
description: The requested resource was not found.
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
schemas:
Error:
type: object
properties:
pc_status:
type: integer
error:
type: string