WebCrawlerAPI

WebCrawler API

REST API on api.webcrawlerapi.com for asynchronous multi-page crawl jobs (POST /v1/crawl, GET /v1/job/{id}, cancel, urls, combined markdown, webhook resend), single-page scraping (POST /v2/scrape sync or ?async=true, GET /v2/scrape/{id}) with output_formats, main_content_only, clean_selectors, max_age caching and prompt + response_schema structured outputs, scheduled change-detection Feeds (POST /v2/feed, Atom 1.0 and JSON Feed 1.1 renderings, pause/resume/run/delete), the autonomous crawling agent (POST /v1/agent with a required max_spend_usd) and organization usage. 24 operations published as Swagger 2.0 at https://api.webcrawlerapi.com/swagger/doc.json behind a live Swagger UI; Bearer API-key auth.

Documentation GitHub OpenAPI

Operations 24

GET /ping Ping API

POST /v1/agent Run an agent

GET /v1/agent/job/{id} Get agent job info

GET /v1/agent/jobs List agent jobs

GET /v1/auth Check authentication

POST /v1/crawl Create a new crawl job

GET /v1/job/{id} Get job details

PUT /v1/job/{id}/cancel Cancel a job

GET /v1/job/{id}/markdown Get combined markdown for a job

GET /v1/job/{id}/urls Get job URLs

POST /v1/job/{id}/webhook/resend Resend webhook

POST /v2/feed Create a new feed

GET /v2/feed/{id} Get feed details

DELETE /v2/feed/{id} Delete a feed

GET /v2/feed/{id}/json Get feed in JSON format

PUT /v2/feed/{id}/pause Pause a feed

PUT /v2/feed/{id}/resume Resume a paused feed

GET /v2/feed/{id}/rss Get feed in Atom/RSS format

PUT /v2/feed/{id}/run Force run a feed

POST /v2/feed/{id}/webhook/resend Resend feed webhook

GET /v2/feeds List all feeds

GET /v2/organization/usage Get organization usage

POST /v2/scrape Create V2 scrape job

GET /v2/scrape/{id} Get V2 scrape result

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this

find_apisBrowse and filter every API in the catalog.
get_api_artifactsOne API's artifacts, grouped by type.
get_openapiThe primary OpenAPI for this API.
find_similar_apisAPIs that look like this one.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This API

curl "https://apis.io/api/v1/apis/webcrawler-api"

All apis

curl "https://apis.io/api/v1/apis?limit=25"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

OpenAPI Specification

openapi: 3.0.3
info:
  description: WebCrawler API is a powerful web crawling and data extraction service designed for developers who
    need to transform websites into LLM-ready structured data and RAG (Retrieval-Augmented Generation) pipelines.
    Extract clean markdown, HTML, or structured content from any website with enterprise-grade reliability. Features
    include intelligent content extraction, automatic link discovery, customizable crawling patterns, webhook notifications,
    and seamless integration with AI applications. Perfect for building knowledge bases, training datasets, content
    aggregation, competitive analysis, and powering LLM applications with fresh web data.
  title: WebCrawler API
  termsOfService: https://webcrawlerapi.com/tos
  contact:
    name: API Support
    url: https://webcrawlerapi.com/support
    email: support@webcrawlerapi.com
  license:
    name: Commercial
    url: https://webcrawlerapi.com/tos
  version: '2.0'
servers:
- url: https://api.webcrawlerapi.com
  description: Production
x-original-source: https://api.webcrawlerapi.com/swagger/doc.json
x-original-format: Swagger 2.0 (saved verbatim at openapi/_original/webcrawlerapi-com-swagger.json)
x-converted: '2026-09-19'
x-conversion-note: Mechanical Swagger 2.0 -> OpenAPI 3.0.3 rendition by API Evangelist. Paths, parameters, schemas,
  responses and security are the provider's own; nothing was added. The provider ships no operationIds or tag declarations


# --- truncated at 32 KB (35 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/openapi/webcrawlerapi-com-openapi.yml

WebCrawler API

Operations 24

Documentation

Specifications

Other Resources

Work with this as data

MCP server

Tools for apis

Call it yourself

Get an API key

OpenAPI Specification