Crawlee

Crawlee is an open-source web scraping and crawling library maintained by Apify, providing a unified set of crawler classes, request queues, datasets, and key-value stores for building reliable scrapers. It is available for both JavaScript/TypeScript (Node.js) and Python, offering HTTP, Cheerio, JSDOM, LinkeDOM, Puppeteer, Playwright, and Stagehand crawler implementations along with proxy and session management utilities for production-grade scraping.

Crawlee is listed with 2 APIs on the APIs.io network. Tagged areas include Apache 2.0, Apify, Browser Automation, Crawlers, and Harvesting.

Crawlee’s developer surface includes documentation, engineering blog, changelog, CLI, and 26 more developer resources.

🌐 Visit website 📡 Profile data on GitHub

27.4/100 thin ▬ flat Agent 6/100 agent aware Front door AI 1/6 silent Heartbeat 70/100 alive self hosted · Apache-2.0 Full breakdown ↓
scored 2026-11-13 · rubric v0.28.0 · Show this score on your site →

AccessPaidOpen

2 APIs

Apache 2.0 Apify Browser Automation Crawlers Harvesting JavaScript Node.js Open Source Playwright Puppeteer Python Web Scraping Web

Kin Score

Kin Score How this is scored →
scored 2026-11-13 · rubric v0.28.0

Composite quality — 27.4/100 · thin

Contract Quality 0.0 / 19

Developer Ergonomics 10.0 / 15

Access Clarity 1.0 / 15

Operational Transparency 2.6 / 10

Contract Governance 1.6 / 9

Discoverability 4.8 / 8

Regulatory Posture 0.7 / 15

Open Source Surface 5.0 / 10

Agent readiness — 6/100 · agent aware

Machine-Readable Contract 0 / 18

Agentic Access Contract 0 / 10

Documented Reversibility 0 / 6

MCP Server 0 / 12

Machine-Readable Auth 0 / 10

Idempotency 0 / 9

Stable Error Semantics 0 / 8

Request/Response Examples 0 / 7

Rate-Limit Signaling 7 / 7

Typed Event Surface 0 / 6

Agent Skills 5 / 5

Well-Known Catalog 0 / 4

Consent & Bot Identity 0 / 3

A2A Agent Card 0 / 8

Dry-Run / Simulate Mode 0 / 4

Delegated User Identity 0 / 6

Protected Resource Metadata 0 / 5

Registration Without a Human 0 / 6

Agentic Commerce Well-Known Document 0 / 5

Pagination 5 / 5

Distinct Operations 3 / 3

MCP Tool Quality 6 / 6

Regulatory Posture applies to this provider. Its tags matched the Horizontal (data, software, accessibility, platform) regime, so Regulatory Posture carries 15 points of the composite. If this regime is wrong for your business, say so on your provider repo — the applicability map is public and we will correct it.

Open Source Surface applies to this provider. This product is open source and we read its repository directly, so Open Source Surface carries 10 points of the composite. It is scored from what the repository actually publishes — a security policy, a contribution guide, a release history, a code of conduct — read live from the provider rather than inferred from our own catalog pointers. This facet adds; nothing was taken away to make room for it. An open-source project is not excused from the commercial facets, because exemption would strip it of the points it does earn. If we have the wrong repository, or this product is not open source, say so on your provider repo and we will drop the facet rather than have you publish against it.

Create-or-Update Ergonomics could not be measured. We hold no machine-readable contract for this provider to read, so there is nothing to measure a write surface against. Excluded rather than scored zero: never-measured and measured-empty are different facts. Publishing an OpenAPI is what makes this facet — and several others — scorable at all.

The six quality facets above are damped to 75 points between them, because both conditional facets apply and carry 25 points together. That is why each facet's contribution is shown against a damped maximum: raising a quality facet moves the composite by 75% of its nominal weight, not 100%. The full arithmetic is at apis.io/rating/.

Show this score on your site.

It updates itself when the score changes. Paste this where you link to your API docs:

<a href="https://apis.io/providers/crawlee/"><img src="https://apis.io/badge/crawlee/card.svg" alt="Crawlee on APIs.io — Kin Score by API Evangelist"></a>

Other styles, a README pill and a shields.io endpoint are on the badge page.

Raise your score.

Agent readiness — checks not met, largest first (exact, from this profile):

Machine-Readable Contract +18 pts — api_specs includes an OpenAPI document Ask about this →
MCP Server +12 pts — deployment.mode + probe verdict on the mcp/ manifest, not the pointer; an unprobed slug falls back to authorship Ask about this →
Agentic Access Contract +10 pts — agentic_access artifact present (x-agentic-access), graded by author Ask about this →
Machine-Readable Auth +10 pts — a served openid-configuration/oauth-authorization-server carrying `issuer` (served), else the securitySchemes class in the provider OpenAPIs — bound (mutualTLS, DPoP, Signature) > negotiable (oauth2 authorizationCode/clientCredentials, openIdConnect) > bearer (http bearer, apiKey) Ask about this →
Idempotency +9 pts — OpenAPI mutating operations declare an idempotency key — a header, a request-body property, or a named de-duplication token whose own description says so (verified); else common[].type matches /idempoten/i (documented) Ask about this →
Stable Error Semantics +8 pts — OpenAPI $refs a shared error schema across 4xx/5xx responses (verified), else common[].type includes ErrorCatalog / ErrorCodes / DeclineCodes Ask about this →

11 more unmet checks in the Agent Readiness table above.

Kin Score — facets with the most composite points still available (weight × headroom):

Contract Quality scores 0.0 — up to +25.0 composite points Ask about this →
Access Clarity scores 6.4 — up to +18.7 composite points Ask about this →
Contract Governance scores 18.2 — up to +9.8 composite points Ask about this →

Each check links to what satisfies it. Publish the artifact, or tell us where it already lives, and the score moves on the next scoring run.

Not sure how this applies to your stack? Ask about any one item. Kin Lane reads every question, looks at your profile and your situation, and replies with what a fix would take — scoped to that one area, priced up front, yours whether you ship it or not. Ask a question →

Improve this rating by publishing the missing artifacts — every area above can be raised, and the full rubric is at apis.io/rating/. Every facet and dimension name above is a link: it opens that measurement's own page — what it means, the exact checks that feed it, how the whole catalog distributes on it, and the providers at the top of it. This rating is computed from the provider profile at artifacts.apis.io/crawlee/apis.yml: open an issue on github.com/api-evangelist/crawlee to ask a question or point us at an artifact you publish. Submit an artifact on GitHub — free → Manage your own listing — the Influence plan, $499/mo →

Standards implemented 1

Standards this provider implements or carries. De facto ones are profiled and measured by the API Commons; formal ones come from the standards directory and are linked by the provider's own tags. The evidence column says how each claim was established — not that it was made.

Agent Skills

measured — observed by APIs.io (published)

APIs 2

Individual APIs this provider publishes, each with its own machine-readable definition.

Crawlee JavaScript SDK

The Crawlee JavaScript SDK is a Node.js/TypeScript library for building reliable web scrapers and crawlers. It provides a family of crawler classes - BasicCrawler, HttpCrawler, ...

Crawlee Python SDK

The Crawlee Python SDK is a Python library for building reliable web scrapers and crawlers. It offers BasicCrawler, HttpCrawler, BeautifulSoupCrawler, ParselCrawler, PlaywrightC...

Crawlee API pricing

Pricing Plans 1

Published pricing tiers and plan structures.

Crawlee Plans Pricing

0 plans

PLANS

Rate Limits 1

Documented rate limits and quota policies.

Crawlee Rate Limits

0 limits

RATE LIMITS

Terms of Service 1

What its API terms permit and prohibit, quoted from its own page.

Crawlee

2 quoted statements, read from Crawlee's own page on 2026-11-09.

TERMS

Data License 1

How its data and outputs may be reused, quoted from its own terms.

Crawlee

2 quoted statements, read from Crawlee's own page on 2026-11-09.

LICENSE

FinOps 1

Cost, billing, and metering signals for API financial operations.

Crawlee Finops

FINOPS

Security Posture 1

Authentication, domain security, vulnerability disclosure, and trust-center signals.

Crawlee Domain Security

TLSv1.3 · HSTS · DNSSEC

SECURITY

Resources

Get Started 1

Portal, sign-up, and the first successful call

DeveloperPortal

DeveloperPortal

Documentation 2

Reference material describing how the API behaves

ContributionGuide

ContributionGuide

Documentation

Documentation

Agent Surfaces 2

MCP servers, agent skills, and machine-readable catalogs

LLMsTxt

LLMsTxt

AgentSkill

AgentSkill

Design & Contract 3

Pagination, idempotency, versioning, errors, and events

Lifecycle

Lifecycle

Conventions

Conventions

Conformance

Conformance

Build 5

SDKs, sample code, and the tooling you integrate with

GitHubOrganization

GitHubRepository

Packages

SDKs

CLI

Access & Security 1

Authentication, authorization, and security posture

DomainSecurity

DomainSecurity

Operate 7

Status, limits, changes, and where to get help

RateLimits

IssueTracker

ChangeLog

Discord

Community

ChangeLog

Deprecation

Scroll for all 7

Commercial 5

Pricing, plans, and the legal terms of use

Plans

FinOps

License

UsageTerms

DataLicense

Company 3

The organization behind the API

LinkedIn

Website

Website

Blog

Blog

Other 1

Properties that don't map to a standard resource type

Apify

Apify

Source (apis.yml)

aid: crawlee
name: Crawlee
kind: opensource
description: Crawlee is an open-source web scraping and crawling library maintained by Apify, providing a unified set of crawler
  classes, request queues, datasets, and key-value stores for building reliable scrapers. It is available for both JavaScript/TypeScript
  (Node.js) and Python, offering HTTP, Cheerio, JSDOM, LinkeDOM, Puppeteer, Playwright, and Stagehand crawler implementations
  along with proxy and session management utilities for production-grade scraping.
url: https://raw.githubusercontent.com/api-evangelist/crawlee/refs/heads/main/apis.yml
deliveryModel:
  model: self-hosted
  license: Apache-2.0
  license_evidence:
    repo: apify/crawlee
    spdx: Apache-2.0
    stars: 25890
  open_source: true
  commercial: false
  callable_host: false
  label: Self-hosted open source · you run it yourself
  confidence: medium
  source:
  - repository-license
  generated: '2026-09-25'
  method: derived
accessModel:
  pricing: paid
  onboarding: open
  trial: false
  try_now: false
  public: true
  gated: none
  gated_source:
  - onboarding
  label: Paid · Open access
  confidence: medium
  source:
  - finops
  - rate-limits
  - security
  generated: '2026-11-13'
  method: derived
image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/icons/crawlee.png
x-docs-platform: docusaurus
x-docs-platform-confidence: medium
x-docs-platform-checked: 2026-09-24
tags:
- Apache 2.0
- Apify
- Browser Automation
- Crawlers
- Harvesting
- JavaScript
- Node.js
- Open Source
- Playwright
- Puppeteer
- Python
- Web Scraping
- Web
tags_raw:
- Apache 2.0
- Apify
- Browser Automation
- Crawlers
- Harvesting
- JavaScript
- Node.js
- Open Source
- Playwright
- Puppeteer
- Python
- Scraping
- Web
- Open-Source
created: '2025-02-08'
modified: '2026-09-07'
specificationVersion: '0.23'
type: Index
access: Public
position: Producing
apis:
- aid: crawlee:crawlee-javascript-sdk
  name: Crawlee JavaScript SDK
  description: The Crawlee JavaScript SDK is a Node.js/TypeScript library for building reliable web scrapers and crawlers.
    It provides a family of crawler classes - BasicCrawler, HttpCrawler, CheerioCrawler, JSDOMCrawler, LinkeDOMCrawler, PuppeteerCrawler,
    PlaywrightCrawler, and AdaptivePlaywrightCrawler - along with shared infrastructure for AutoscaledPool resource management,
    proxy rotation, session pooling, RequestQueue task queuing, Dataset result storage, and KeyValueStore unstructured data
    persistence. Crawlee handles retries, error recovery, request fingerprinting, and statistics tracking out of the box,
    allowing developers to focus on extraction logic.
  humanURL: https://crawlee.dev/js
  properties:
  - type: Documentation
    url: https://crawlee.dev/js
  - type: Reference
    url: https://crawlee.dev/js/api
  - type: GettingStarted
    url: https://crawlee.dev/js/docs/quick-start
  - type: GitHubRepository
    url: https://github.com/apify/crawlee
  - type: NpmPackage
    url: https://www.npmjs.com/package/crawlee
  tags:
  - Browser Automation
  - Cheerio
  - JavaScript
  - Node.js
  - Playwright
  - Puppeteer
  - Web Scraping
  - TypeScript
  tags_raw:
  - Browser Automation
  - Cheerio
  - JavaScript
  - Node.js
  - Playwright
  - Puppeteer
  - Scraping
  - TypeScript
- aid: crawlee:crawlee-python-sdk
  name: Crawlee Python SDK
  description: The Crawlee Python SDK is a Python library for building reliable web scrapers and crawlers. It offers BasicCrawler,
    HttpCrawler, BeautifulSoupCrawler, ParselCrawler, PlaywrightCrawler, and Adaptive crawlers built on top of asyncio, along
    with shared infrastructure for proxy rotation, session pooling, RequestQueue, Dataset, and KeyValueStore. The Python SDK
    targets data engineers and Python developers who want the same crawler ergonomics as the JavaScript version but inside
    the Python ecosystem.
  humanURL: https://crawlee.dev/python
  properties:
  - type: Documentation
    url: https://crawlee.dev/python
  - type: Reference
    url: https://crawlee.dev/python/api
  - type: GettingStarted
    url: https://crawlee.dev/python/docs/quick-start
  - type: GitHubRepository
    url: https://github.com/apify/crawlee-python
  - type: PyPiPackage
    url: https://pypi.org/project/crawlee/
  tags:
  - Async I/O
  - BeautifulSoup
  - Browser Automation
  - Parsel
  - Playwright
  - Python
  - Web Scraping
  tags_raw:
  - Asyncio
  - BeautifulSoup
  - Browser Automation
  - Parsel
  - Playwright
  - Python
  - Scraping
common:
- type: Plans
  url: plans/crawlee-plans-pricing.yml
- type: RateLimits
  url: rate-limits/crawlee-rate-limits.yml
- type: FinOps
  url: finops/crawlee-finops.yml
- type: IssueTracker
  url: https://github.com/apify/crawlee/issues
- type: ContributionGuide
  url: https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md
- type: DomainSecurity
  url: security/crawlee-domain-security.yml
- type: LinkedIn
  url: https://www.linkedin.com/company/apify
- type: Website
  url: https://crawlee.dev/
- type: Documentation
  url: https://crawlee.dev/
- type: GitHubOrganization
  url: https://github.com/apify
- type: GitHubRepository
  url: https://github.com/apify/crawlee
- type: Blog
  url: https://crawlee.dev/blog
- type: ChangeLog
  url: https://github.com/apify/crawlee/releases
- type: Discord
  url: https://discord.gg/jyEM2PRvMU
- type: Community
  url: https://discord.gg/jyEM2PRvMU
- type: License
  url: https://github.com/apify/crawlee/blob/master/LICENSE.md
- type: Apify
  url: https://apify.com/
- type: DeveloperPortal
  url: https://crawlee.dev/
- type: LLMsTxt
  url: llms/crawlee-llms.txt
- type: Packages
  url: packages/crawlee-packages.yml
- type: SDKs
  url: packages/crawlee-packages.yml
- type: CLI
  url: cli/crawlee-cli.yml
- type: ChangeLog
  url: changelog/crawlee-changelog.yml
- type: Lifecycle
  url: lifecycle/crawlee-lifecycle.yml
- type: Deprecation
  url: lifecycle/crawlee-lifecycle.yml
- type: Conventions
  url: conventions/crawlee-conventions.yml
- type: Conformance
  url: conformance/crawlee-conformance.yml
- type: AgentSkill
  url: skills/_index.yml
- type: UsageTerms
  url: terms/crawlee-terms-of-service.yml
- type: DataLicense
  url: data-license/crawlee-data-license.yml
maintainers:
- FN: Kin Lane
  email: kin@apievangelist.com
x-enrichment:
  date: '2026-09-07'
  status: enriched
  artifacts_added: 12
  pass: local-v3

Crawlee API — frequently asked questions

Does Crawlee have an API?

Yes. APIs.io lists 2 APIs from Crawlee, including Crawlee JavaScript SDK and Crawlee Python SDK.

Is the Crawlee API ready for AI agents?

Crawlee has a Kin Score of 27.4 out of 100 (thin) and an Agent Readiness score of 6.1 (agent-aware), scored by API Evangelist on 2026-11-13.

Work with this as data

Every provider here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for providers

9 MCP tools reach this

find_providersBrowse and filter every provider in the catalog.
get_provider_artifactsEvery artifact this provider publishes, grouped by type.
get_provider_operationsEvery operation across all of their OpenAPIs — one call instead of parsing every spec.
get_provider_toolsEvery MCP tool they ship, with the operation each wraps.
get_provider_evidenceHow each part of their score was established. Free — the basis for a claim should not sit behind it.
get_provider_ratingPRO — composite, band, trend and facet scores.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This provider

curl "https://apis.io/api/v1/providers/crawlee"

All providers

curl "https://apis.io/api/v1/providers?limit=25"

Every operation they expose

curl "https://apis.io/api/v1/providers/crawlee/operations?limit=25"

How their score was established

curl "https://apis.io/api/v1/providers/crawlee/evidence"

Calls without X-API-Key need no key. Calls with it need a paid plan — get a key.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.