Crawlbase Crawling API

openapi: 3.0.3
info:
  title: Crawlbase API
  description: >-
    Crawlbase (formerly ProxyCrawl) is a web crawling and scraping platform.
    A single REST host, https://api.crawlbase.com, exposes several products,
    all authenticated with a `token` query parameter: the Crawling API (fetch
    any URL through a rotating proxy network, optionally rendered in headless
    Chrome), the Scraper API (ready-made structured extractors for popular
    sites), the Cloud Storage API (retrieve/list/delete previously stored
    crawls), the Screenshots API (rendered page captures), and the Leads API
    (domain email discovery).

    Grounding note: paths, methods, and the token query-param auth model are
    confirmed from Crawlbase's public documentation. Request query parameters
    are modeled from the documented parameter reference. Response bodies are
    largely raw upstream content (HTML, Markdown, JSON, or images) or, for the
    Scraper API, provider-specific JSON whose exact fields vary per scraper and
    are therefore modeled loosely rather than enumerated. Treat response schemas
    as illustrative.
  version: '1.0'
  contact:
    name: Crawlbase
    url: https://crawlbase.com
servers:
  - url: https://api.crawlbase.com
    description: Crawlbase API host (all products share this host)
security:
  - tokenAuth: []
tags:
  - name: Crawling API
    description: Fetch any URL through the rotating proxy network, optionally rendered.
  - name: Scraper API
    description: Ready-made structured-data extractors for supported sites (legacy).
  - name: Storage API
    description: Retrieve, list, and delete previously stored crawls.
  - name: Screenshots API
    description: Rendered page screenshots in headless Chrome (legacy).
  - name: Leads API
    description: Publicly visible email discovery for a domain (legacy).
paths:
  /:
    get:
      operationId: crawl
      tags:
        - Crawling API
      summary: Crawl a URL
      description: >-
        Fetches the target `url` through Crawlbase's proxy network and returns
        the page. Use the Normal (TCP) token for static HTML/JSON, or the
        JavaScript token to render the page in headless Chrome before returning
        it. Crawlbase status is surfaced in the `pc_status` response header;
        requests that do not return `pc_status: 200` are not charged.
      parameters:
        - $ref: '#/components/parameters/Url'
        - $ref: '#/components/parameters/Format'
        - $ref: '#/components/parameters/Country'
        - $ref: '#/components/parameters/Device'
        - $ref: '#/components/parameters/UserAgent'
        - $ref: '#/components/parameters/Scraper'
        - name: autoparse
          in: query
          required: false
          description: Auto-detect the page type and return parsed JSON.
          schema:
            type: boolean
        - name: page_wait
          in: query
          required: false
          description: Milliseconds to wait after page load before capture (JavaScript token only).
          schema:
            type: integer
        - name: ajax_wait
          in: query
          required: false
          description: Wait for AJAX / network idle before capture (JavaScript token only).
          schema:
            type: boolean
        - name: css_click_selector
          in: query
          required: false
          description: CSS selector to click before capture (JavaScript token only).
          schema:
            type: string
        - name: scroll
          in: query
          required: false
          description: Scroll toward the bottom of the page before capture (JavaScript token only).
          schema:
            type: boolean
        - name: scroll_interval
          in: query
          required: false
          description: Maximum seconds to scroll (default 10, max 60).
          schema:
            type: integer
        - name: screenshot
          in: query
          required: false
          description: Capture a JPEG screenshot of the rendered page (JavaScript token only).
          schema:
            type: boolean
        - name: get_headers
          in: query
          required: false
          description: Return the target's response headers alongside the body.
          schema:
            type: boolean
        - name: get_cookies
          in: query
          required: false
          description: Return the target's Set-Cookie values.
          schema:
            type: boolean
        - name: cookies_session
          in: query
          required: false
          description: Sticky session identifier to reuse cookies across requests.
          schema:
            type: string
        - name: store
          in: query
          required: false
          description: Persist the response in Crawlbase Cloud Storage.
          schema:
            type: boolean
        - name: async
          in: query
          required: false
          description: Queue the request and return immediately with a request id (rid).
          schema:
            type: boolean
        - name: callback
          in: query
          required: false
          description: Webhook URL that Crawlbase posts async results to.
          schema:
            type: string
            format: uri
      responses:
        '200':
          description: >-
            The crawled page. Body is raw upstream content (HTML, Markdown, JSON,
            or image) depending on `format`, `scraper`, and `screenshot`.
            Crawlbase-specific metadata (pc_status, original_status, rid, url) is
            returned in response headers, or inline when `format=json`.
          content:
            text/html:
              schema:
                type: string
            application/json:
              schema:
                type: object
                additionalProperties: true
        '401':
          $ref: '#/components/responses/Unauthorized'
    post:
      operationId: crawlPost
      tags:
        - Crawling API
      summary: Crawl a URL (POST)
      description: >-
        Same as the GET crawl operation, but the request payload (for example,
        POST form data to submit to the target) is sent in the request body.
        The `token` and `url` remain query parameters.
      parameters:
        - $ref: '#/components/parameters/Url'
        - $ref: '#/components/parameters/Format'
        - $ref: '#/components/parameters/Country'
        - $ref: '#/components/parameters/Device'
      requestBody:
        required: false
        content:
          application/x-www-form-urlencoded:
            schema:
              type: object
              additionalProperties: true
      responses:
        '200':
          description: The crawled page.
          content:
            text/html:
              schema:
                type: string
            application/json:
              schema:
                type: object
                additionalProperties: true
        '401':
          $ref: '#/components/responses/Unauthorized'
  /scraper:
    get:
      operationId: scrape
      tags:
        - Scraper API
      summary: Scrape a supported site to structured JSON
      description: >-
        Applies a named `scraper` to the target `url` and returns provider-specific
        structured JSON (for example, amazon-product-details, google-serp,
        linkedin-company). Documented by Crawlbase as a legacy endpoint; the
        Crawling API's `&scraper=` parameter offers the same extractors.
      parameters:
        - $ref: '#/components/parameters/Url'
        - name: scraper
          in: query
          required: true
          description: Name of the scraper/extractor to apply (see the Crawlbase scraper catalog).
          schema:
            type: string
        - $ref: '#/components/parameters/Country'
        - name: javascript
          in: query
          required: false
          description: Render the page in Chrome before scraping (requires the JavaScript token).
          schema:
            type: boolean
        - name: premium
          in: query
          required: false
          description: Route through the premium residential proxy network.
          schema:
            type: boolean
      responses:
        '200':
          description: Structured JSON for the scraped page. Fields vary per scraper.
          content:
            application/json:
              schema:
                type: object
                additionalProperties: true
        '401':
          $ref: '#/components/responses/Unauthorized'
  /storage:
    get:
      operationId: getStoredPage
      tags:
        - Storage API
      summary: Retrieve a stored page
      description: >-
        Returns a previously stored crawl by `rid` or by `url`. When queried by
        `url`, the most recent stored version is returned. Exactly one of `rid`
        or `url` is required.
      parameters:
        - name: rid
          in: query
          required: false
          description: The request id of a stored crawl.
          schema:
            type: string
        - name: url
          in: query
          required: false
          description: The URL-encoded target URL whose most recent stored version to return.
          schema:
            type: string
        - $ref: '#/components/parameters/Format'
      responses:
        '200':
          description: The stored page.
          content:
            text/html:
              schema:
                type: string
            application/json:
              schema:
                type: object
                additionalProperties: true
        '401':
          $ref: '#/components/responses/Unauthorized'
        '404':
          $ref: '#/components/responses/NotFound'
    delete:
      operationId: deleteStoredPage
      tags:
        - Storage API
      summary: Delete a stored page
      description: Deletes a single stored crawl identified by its `rid`.
      parameters:
        - name: rid
          in: query
          required: true
          description: The request id of the stored crawl to delete.
          schema:
            type: string
      responses:
        '200':
          description: Deletion confirmation.
          content:
            application/json:
              schema:
                type: object
                additionalProperties: true
        '401':
          $ref: '#/components/responses/Unauthorized'
        '404':
          $ref: '#/components/responses/NotFound'
  /storage/bulk:
    post:
      operationId: getStoredPagesBulk
      tags:
        - Storage API
      summary: Bulk retrieve stored pages
      description: Fetches up to 100 stored crawls in a single request by a list of request ids.
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                rids:
                  type: array
                  items:
                    type: string
      responses:
        '200':
          description: A batch of stored pages.
          content:
            application/json:
              schema:
                type: object
                additionalProperties: true
        '401':
          $ref: '#/components/responses/Unauthorized'
  /storage/bulk_delete:
    post:
      operationId: deleteStoredPagesBulk
      tags:
        - Storage API
      summary: Bulk delete stored pages
      description: Deletes multiple stored crawls in a single request by a list of request ids.
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                rids:
                  type: array
                  items:
                    type: string
      responses:
        '200':
          description: Deletion confirmation.
          content:
            application/json:
              schema:
                type: object
                additionalProperties: true
        '401':
          $ref: '#/components/responses/Unauthorized'
  /storage/rids:
    get:
      operationId: listStoredRids
      tags:
        - Storage API
      summary: List stored request ids
      description: Lists stored request ids with scroll-based pagination.
      parameters:
        - name: limit
          in: query
          required: false
          description: Maximum request ids to return per page (cap 10000).
          schema:
            type: integer
        - name: scroll
          in: query
          required: false
          description: Enable scroll pagination; the response returns a scroll_id to continue.
          schema:
            type: boolean
      responses:
        '200':
          description: A page of request ids.
          content:
            application/json:
              schema:
                type: object
                additionalProperties: true
        '401':
          $ref: '#/components/responses/Unauthorized'
  /storage/total_count:
    get:
      operationId: getStoredTotalCount
      tags:
        - Storage API
      summary: Total stored page count
      description: Returns the total number of pages currently in Cloud Storage.
      responses:
        '200':
          description: The total stored count.
          content:
            application/json:
              schema:
                type: object
                properties:
                  totalCount:
                    type: integer
        '401':
          $ref: '#/components/responses/Unauthorized'
  /screenshots:
    get:
      operationId: screenshot
      tags:
        - Screenshots API
      summary: Capture a screenshot of a URL
      description: >-
        Renders the target `url` in headless Chrome and returns a screenshot.
        Requires the JavaScript token. Documented by Crawlbase as a legacy
        endpoint (closed to new sign-ups on 2024-11-01).
      parameters:
        - $ref: '#/components/parameters/Url'
        - name: mode
          in: query
          required: false
          description: viewport (visible area) or fullpage (entire scrollable page).
          schema:
            type: string
            enum:
              - viewport
              - fullpage
            default: viewport
        - name: format
          in: query
          required: false
          description: Image output format.
          schema:
            type: string
            enum:
              - png
              - jpeg
            default: png
        - name: width
          in: query
          required: false
          description: Viewport width in pixels.
          schema:
            type: integer
            default: 1280
        - name: height
          in: query
          required: false
          description: Viewport height in pixels.
          schema:
            type: integer
            default: 800
        - $ref: '#/components/parameters/Device'
        - name: store
          in: query
          required: false
          description: Persist the capture in Cloud Storage.
          schema:
            type: boolean
      responses:
        '200':
          description: The rendered screenshot image.
          content:
            image/png:
              schema:
                type: string
                format: binary
            image/jpeg:
              schema:
                type: string
                format: binary
        '401':
          $ref: '#/components/responses/Unauthorized'
  /leads:
    get:
      operationId: getLeads
      tags:
        - Leads API
      summary: Discover emails for a domain
      description: >-
        Returns publicly visible email addresses associated with a bare `domain`,
        with the source URLs where each was found. Billed at one credit per ten
        emails. Documented by Crawlbase as closed to new sign-ups since
        2024-10-01.
      parameters:
        - name: domain
          in: query
          required: true
          description: Bare domain (no scheme or path), for example slack.com.
          schema:
            type: string
        - name: limit
          in: query
          required: false
          description: Maximum emails to return (default 10).
          schema:
            type: integer
            default: 10
      responses:
        '200':
          description: Discovered emails with source URLs and available metadata.
          content:
            application/json:
              schema:
                type: object
                additionalProperties: true
        '401':
          $ref: '#/components/responses/Unauthorized'
components:
  securitySchemes:
    tokenAuth:
      type: apiKey
      in: query
      name: token
      description: >-
        Crawlbase authentication token passed as the `token` query parameter.
        Each account has two tokens: a Normal (TCP) token for static content and
        a JavaScript token for headless-Chrome rendering.
  parameters:
    Url:
      name: url
      in: query
      required: true
      description: Fully URL-encoded target URL, including http:// or https://.
      schema:
        type: string
    Format:
      name: format
      in: query
      required: false
      description: Response format - html (default), json, or md (Markdown).
      schema:
        type: string
        enum:
          - html
          - json
          - md
        default: html
    Country:
      name: country
      in: query
      required: false
      description: Two-letter ISO 3166 country code to geolocate the request (for example US, DE, GB).
      schema:
        type: string
    Device:
      name: device
      in: query
      required: false
      description: Device profile - desktop (default), tablet, or mobile.
      schema:
        type: string
        enum:
          - desktop
          - tablet
          - mobile
        default: desktop
    UserAgent:
      name: user_agent
      in: query
      required: false
      description: Override the User-Agent header sent to the target.
      schema:
        type: string
    Scraper:
      name: scraper
      in: query
      required: false
      description: Name of a built-in extractor to apply to the crawled page (returns parsed JSON).
      schema:
        type: string
  responses:
    Unauthorized:
      description: Missing or invalid token.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
    NotFound:
      description: The requested resource was not found.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
  schemas:
    Error:
      type: object
      properties:
        pc_status:
          type: integer
        error:
          type: string