# WebCrawler API

**Canonical:** https://apis.io/apis/webcrawlerapi-com/webcrawler-api/  
**Provider:** WebCrawlerAPI — https://apis.io/providers/webcrawlerapi-com/  
**Base URL:** https://api.webcrawlerapi.com  
**Documentation:** https://webcrawlerapi.com/docs/getting-started

WebCrawler API is one of 3 APIs that [WebCrawlerAPI](https://apis.io/providers/webcrawlerapi-com/) publishes on the [APIs.io](https://apis.io/) network, described by a machine-readable OpenAPI specification. Tagged areas include Web Scraping, Web Crawling, Data Extraction, Markdown, and Web Feeds. The published artifact set on APIs.io includes an OpenAPI specification, API documentation, an API reference, authentication docs, and rate-limit docs.

REST API on api.webcrawlerapi.com for asynchronous multi-page crawl jobs (POST /v1/crawl, GET /v1/job/{id}, cancel, urls, combined markdown, webhook resend), single-page scraping (POST /v2/scrape sync or ?async=true, GET /v2/scrape/{id}) with output_formats, main_content_only, clean_selectors, max_age caching and prompt + response_schema structured outputs, scheduled change-detection Feeds (POST /v2/feed, Atom 1.0 and JSON Feed 1.1 renderings, pause/resume/run/delete), the autonomous crawling agent (POST /v1/agent with a required max_spend_usd) and organization usage. 24 operations published as Swagger 2.0 at https://api.webcrawlerapi.com/swagger/doc.json behind a live Swagger UI; Bearer API-key auth.

## Operations (24)

| Method | Path | Summary |
|---|---|---|
| GET | `/ping` | Ping API |
| POST | `/v1/agent` | Run an agent |
| GET | `/v1/agent/job/{id}` | Get agent job info |
| GET | `/v1/agent/jobs` | List agent jobs |
| GET | `/v1/auth` | Check authentication |
| POST | `/v1/crawl` | Create a new crawl job |
| GET | `/v1/job/{id}` | Get job details |
| PUT | `/v1/job/{id}/cancel` | Cancel a job |
| GET | `/v1/job/{id}/markdown` | Get combined markdown for a job |
| GET | `/v1/job/{id}/urls` | Get job URLs |
| POST | `/v1/job/{id}/webhook/resend` | Resend webhook |
| POST | `/v2/feed` | Create a new feed |
| GET | `/v2/feed/{id}` | Get feed details |
| DELETE | `/v2/feed/{id}` | Delete a feed |
| GET | `/v2/feed/{id}/json` | Get feed in JSON format |
| PUT | `/v2/feed/{id}/pause` | Pause a feed |
| PUT | `/v2/feed/{id}/resume` | Resume a paused feed |
| GET | `/v2/feed/{id}/rss` | Get feed in Atom/RSS format |
| PUT | `/v2/feed/{id}/run` | Force run a feed |
| POST | `/v2/feed/{id}/webhook/resend` | Resend feed webhook |
| GET | `/v2/feeds` | List all feeds |
| GET | `/v2/organization/usage` | Get organization usage |
| POST | `/v2/scrape` | Create V2 scrape job |
| GET | `/v2/scrape/{id}` | Get V2 scrape result |

## Machine-readable artifacts (16)

- **OpenAPI** — https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/openapi/webcrawlerapi-com-openapi.yml
- **OpenAPI** — https://api.webcrawlerapi.com/swagger/doc.json
- **OpenAPI** — https://api.webcrawlerapi.com/openapi/agent.json
- **Documentation** — https://webcrawlerapi.com/docs/getting-started
- **APIReference** — https://api.webcrawlerapi.com/swagger/index.html
- **APIReference** — https://webcrawlerapi.com/docs/api/crawl
- **Overlay** — https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/overlays/webcrawlerapi-com-openapi-overlay.yaml
- **Authentication** — https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/authentication/webcrawlerapi-com-authentication.yml
- **Conventions** — https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/conventions/webcrawlerapi-com-conventions.yml
- **ErrorCatalog** — https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/errors/webcrawlerapi-com-problem-types.yml
- **DataModel** — https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/data-model/webcrawlerapi-com-data-model.yml
- **Webhooks** — https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/asyncapi/webcrawlerapi-com-webhooks.yml
- **RateLimits** — https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/rate-limits/webcrawlerapi-com-rate-limits.yml
- **Postman** — https://www.postman.com/webcrawlerapi/webcrawlerapi-public-workspace/collection/xz2fs5u/webcrawlerapi
- **AgentSkill** — https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/skills/_index.yml
- **APIsJSON** — https://raw.githubusercontent.com/api-evangelist/webcrawlerapi-com/refs/heads/main/apis.yml

## Other WebCrawlerAPI APIs (2)

- [WebCrawlerAPI MCP Server](https://apis.io/apis/webcrawlerapi-com/webcrawlerapi-mcp-server/)
- [WebCrawlerAPI Agent Card](https://apis.io/apis/webcrawlerapi-com/webcrawlerapi-a2a-agent-card/)

## Tags

Web Scraping, Web Crawling, Data Extraction, Markdown, Web Feeds, Structured Data, AI Agents

---

Profiled by [API Evangelist](https://apievangelist.com) and published on [APIs.io](https://apis.io/apis/webcrawlerapi-com/webcrawler-api/). The API's provider profile, Kin Score and agent-readiness rating are at https://apis.io/providers/webcrawlerapi-com/.