WebCrawlerAPI

WebCrawlerAPI Jobs API

The Jobs API from WebCrawlerAPI — 6 operation(s) for jobs.

GitHub OpenAPI

Operations 6

POST /v1/crawl Create a new crawl job #
GET /v1/job/{id} Get job details #
PUT /v1/job/{id}/cancel Cancel a job #
GET /v1/job/{id}/markdown Get combined markdown for a job #
GET /v1/job/{id}/urls Get job URLs #
POST /v1/job/{id}/webhook/resend Resend webhook #

Documentation

📖
Documentation
https://webcrawlerapi.com/docs/getting-started
📖
APIReference
https://api.webcrawlerapi.com/swagger/index.html
📖
APIReference
https://webcrawlerapi.com/docs/api/crawl
📖
Authentication
https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/authentication/webcrawlerapi-com-authentication.yml
📖
RateLimits
https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/rate-limits/webcrawlerapi-com-rate-limits.yml

Specifications

⚙
OpenAPI
https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/openapi/webcrawlerapi-com-jobs-api-openapi.yml

Other Resources

🔗
Conventions
https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/conventions/webcrawlerapi-com-conventions.yml
🔗
ErrorCatalog
https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/errors/webcrawlerapi-com-problem-types.yml
🔗
DataModel
https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/data-model/webcrawlerapi-com-data-model.yml
🔗
Webhooks
https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/asyncapi/webcrawlerapi-com-webhooks.yml
🔗
Postman
https://www.postman.com/webcrawlerapi/webcrawlerapi-public-workspace/collection/xz2fs5u/webcrawlerapi

Work with this as data

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this
  • find_apisBrowse and filter every API in the catalog.
  • get_api_artifactsOne API's artifacts, grouped by type.
  • get_openapiThe primary OpenAPI for this API.
  • find_similar_apisAPIs that look like this one.
  • apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
  • resolveTurn a domain, URL or GitHub org into the provider it belongs to.
  • find_cohortsEvery scored population of providers in the catalog.
All 92 tools →

Call it yourself

curl for this page
This API 
curl "https://apis.io/api/v1/apis/webcrawlerapi-com:webcrawlerapi-com-jobs-api"
All apis 
curl "https://apis.io/api/v1/apis?limit=25"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

OpenAPI Specification

webcrawlerapi-com-jobs-api-openapi.yml Raw ↑ 
openapi: 3.2.0
info:
  description: WebCrawler API is a powerful web crawling and data extraction service designed for developers who need to transform websites into LLM-ready structured data and RAG (Retrieval-Augmented Generation) pipelines.
  title: WebCrawler Jobs API
  termsOfService: https://webcrawlerapi.com/tos
  contact:
    name: API Support
    url: https://webcrawlerapi.com/support
    email: support@webcrawlerapi.com
  license:
    name: Commercial
    url: https://webcrawlerapi.com/tos
  version: '2.0'
servers:
- url: https://api.webcrawlerapi.com
  description: Production
tags:
- name: Jobs
paths:
  /v1/crawl:
    post:
      security:
      - ApiKeyAuth: []
      description: Creates a new web crawling job with specified parameters. This endpoint crawls websites and extracts content based on configuration.
      tags:
      - Jobs
      summary: Create a new crawl job
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/PostJobRequest'
        description: Job parameters
        required: true
      responses:
        '200':
          description: Job created successfully
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
        '400':
          description: Invalid request
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
        '402':
          description: Insufficient balance
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
        '403':
          description: Organization suspended or URL blocked
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
      operationId: postV1Crawl
      x-operation-id-source: derived
  /v1/job/{id}:
    get:
      security:
      - ApiKeyAuth: []
      description: Retrieves the details and status of a specific job including all items
      tags:
      - Jobs
      summary: Get job details
      parameters:
      - description: Job ID
        name: id
        in: path
        required: true
        schema:
          type: string
      - description: Include links found on pages
        name: include_links
        in: query
        schema:
          type: boolean
      responses:
        '200':
          description: Job details
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/CrawlJobView'
        '401':
          description: Access denied
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
        '404':
          description: Job not found
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
      operationId: getV1JobById
      x-operation-id-source: derived
  /v1/job/{id}/cancel:
    put:
      security:
      - ApiKeyAuth: []
      description: Cancels a running or pending job
      tags:
      - Jobs
      summary: Cancel a job
      parameters:
      - description: Job ID
        name: id
        in: path
        required: true
        schema:
          type: string
      responses:
        '200':
          description: Job cancelled successfully
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
        '401':
          description: Access denied
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
        '404':
          description: Job not found
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
      operationId: putV1JobByIdCancel
      x-operation-id-source: derived
  /v1/job/{id}/markdown:
    get:
      security:
      - ApiKeyAuth: []
      description: Downloads all successful markdown job items, combines them with separators, caches and serves as single file
      tags:
      - Jobs
      summary: Get combined markdown for a job
      parameters:
      - description: Job ID
        name: id
        in: path
        required: true
        schema:
          type: string
      responses:
        '200':
          description: Combined markdown content
          content:
            text/markdown:
              schema:
                type: string
        '400':
          description: Job is not a markdown type
          content:
            text/markdown:
              schema:
                type: object
                additionalProperties:
                  type: string
        '401':
          description: Access denied
          content:
            text/markdown:
              schema:
                type: object
                additionalProperties:
                  type: string
        '404':
          description: Job not found or no markdown content
          content:
            text/markdown:
              schema:
                type: object
                additionalProperties:
                  type: string
        '422':
          description: Job not finished
          content:
            text/markdown:
              schema:
                type: object
                additionalProperties:
                  type: string
        '500':
          description: Internal server error
          content:
            text/markdown:
              schema:
                type: object
                additionalProperties:
                  type: string
      operationId: getV1JobByIdMarkdown
      x-operation-id-source: derived
  /v1/job/{id}/urls:
    get:
      security:
      - ApiKeyAuth: []
      description: Retrieves all URLs discovered by a crawl job with path clustering information
      tags:
      - Jobs
      summary: Get job URLs
      parameters:
      - description: Job ID
        name: id
        in: path
        required: true
        schema:
          type: string
      responses:
        '200':
          description: URLs and clusters
          content:
            application/json:
              schema:
                type: object
                additionalProperties: true
        '401':
          description: Access denied
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
        '404':
          description: Job not found
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
      operationId: getV1JobByIdUrls
      x-operation-id-source: derived
  /v1/job/{id}/webhook/resend:
    post:
      security:
      - ApiKeyAuth: []
      description: Resends the webhook notification for a completed job
      tags:
      - Jobs
      summary: Resend webhook
      parameters:
      - description: Job ID
        name: id
        in: path
        required: true
        schema:
          type: string
      responses:
        '200':
          description: Webhook resend status
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/WebhookResendResponse'
        '400':
          description: Job has no webhook URL
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
        '404':
          description: Job not found
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  type: string
      operationId: postV1JobByIdWebhookResend
      x-operation-id-source: derived
components:
  schemas:
    PostJobRequest:
      type: object
      properties:
        blacklist_regexp:
          type: string
          example: .*\/admin\/.*
        items_limit:
          type: integer
          example: 10
        keep_query_params:
          type: boolean
          example: false
        main_content_only:
          type: boolean
          example: false
        max_age:
          type: integer
          example: 604800
        max_depth:
          type: integer
          example: 2
        output_formats:
          type: array
          items:
            type: string
          example:
          - '["markdown"'
          - '"html"]'
        respect_robots_txt:
          type: boolean
          example: false
        scrape_type:
          type: string
          example: markdown
        url:
          type: string
          example: https://example.com
        webhook_url:
          type: string
          example: https://your-domain.com/webhook
        whitelist_regexp:
          type: string
          example: .*\/blog\/.*
    CrawlJobItemView:
      type: object
      properties:
        cleaned_content_url:
          type: string
        cost:
          type: number
        created_at:
          type: string
        error:
          type: string
        error_code:
          type: string
        id:
          type: string
        job_id:
          type: string
        last_error:
          type: string
        links:
          type: array
          items:
            type: string
        markdown_content_url:
          type: string
        original_url:
          type: string
        page_status_code:
          type: integer
        raw_content_url:
          type: string
        referred_url:
          type: string
        status:
          type: string
        title:
          type: string
        updated_at:
          type: string
    WebhookResendResponse:
      type: object
      properties:
        job_id:
          type: string
        message:
          type: string
        status_code:
          type: integer
        success:
          type: boolean
        webhook_error:
          type: string
        webhook_url:
          type: string
    CrawlJobView:
      type: object
      properties:
        allow_subdomains:
          type: boolean
        blacklist_regexp:
          type: string
        created_at:
          type: string
        finished_at:
          type: string
        id:
          type: string
        items_limit:
          type: integer
        job_items:
          type: array
          items:
            $ref: '#/components/schemas/CrawlJobItemView'
        max_depth:
          type: integer
        org_id:
          type: string
        output_formats:
          type: array
          items:
            type: string
        recommended_pull_delay_ms:
          type: integer
        respect_robots_txt:
          type: boolean
        scrape_type:
          type: string
        status:
          type: string
        updated_at:
          type: string
        url:
          type: string
        webhook_url:
          type: string
        whitelist_regexp:
          type: string
  securitySchemes:
    ApiKeyAuth:
      description: 'API key for authentication. Format: Bearer {api_key}'
      type: apiKey
      name: Authorization
      in: header
x-original-source: https://api.webcrawlerapi.com/swagger/doc.json
x-original-format: Swagger 2.0 (saved verbatim at openapi/_original/webcrawlerapi-com-swagger.json)
x-converted: '2026-09-19'
x-conversion-note: Mechanical Swagger 2.0 -> OpenAPI 3.0.3 rendition by API Evangelist. Paths, parameters, schemas, responses and security are the provider's own; nothing was added. The provider ships no operationIds or tag declarations - API Evangelist supplies those in overlays/webcrawlerapi-com-openapi-overlay.yaml.