Crawlee website screenshot

Crawlee

Crawlee is an open-source web scraping and crawling library maintained by Apify, providing a unified set of crawler classes, request queues, datasets, and key-value stores for building reliable scrapers. It is available for both JavaScript/TypeScript (Node.js) and Python, offering HTTP, Cheerio, JSDOM, LinkeDOM, Puppeteer, Playwright, and Stagehand crawler implementations along with proxy and session management utilities for production-grade scraping.

Crawlee publishes 2 APIs on the APIs.io network. Tagged areas include Apache 2.0, Apify, Browser Automation, Crawlers, and Harvesting.

Crawlee’s developer surface includes documentation, engineering blog, changelog, and 9 more developer resources.

🌐 Visit website 📡 Source on GitHub
23.7/100 emerging ▬ flat Agent 3/100 human only Full breakdown ↓
scored 2026-07-28 · rubric v0.6
AccessFreemium
2 APIs
Apache 2.0ApifyBrowser AutomationCrawlersHarvestingJavaScriptNode.jsOpen SourcePlaywrightPuppeteerPythonScrapingWeb

Kin Score

Kin Score Kin Score How this is scored →
scored 2026-07-28 · rubric v0.6
Composite quality — 23.7/100 · emerging
Contract Quality 0.0 / 25
Developer Ergonomics 3.0 / 20
Commercial Clarity 7.9 / 20
Operational Transparency 6.8 / 13
Governance 0.0 / 12
Discoverability 5.9 / 10
Agent readiness — 3/100 · human only
Machine-Readable Contract 0 / 18
Agentic Access Contract 0 / 10
MCP Server 0 / 12
Machine-Readable Auth 0 / 10
Idempotency 0 / 9
Stable Error Semantics 0 / 8
Request/Response Examples 0 / 7
Rate-Limit Signaling 7 / 7
Typed Event Surface 0 / 6
Agent Skills 0 / 5
Well-Known Catalog 0 / 4
Consent & Bot Identity 0 / 3
A2A Agent Card 0 / 8
Dry-Run / Simulate Mode 0 / 4
Improve this rating by publishing the missing artifacts — every area above can be raised, and the full rubric is at apis.io/rating/. This rating is computed from github.com/api-evangelist/crawlee: open an issue to ask a question, or submit a pull request to add artifacts. Want it done for you? Prioritized profiling — $2,500 →

APIs 2

Individual APIs this provider publishes, each with its own machine-readable definition.

Crawlee JavaScript SDK

The Crawlee JavaScript SDK is a Node.js/TypeScript library for building reliable web scrapers and crawlers. It provides a family of crawler classes - BasicCrawler, HttpCrawler, ...

Crawlee Python SDK

The Crawlee Python SDK is a Python library for building reliable web scrapers and crawlers. It offers BasicCrawler, HttpCrawler, BeautifulSoupCrawler, ParselCrawler, PlaywrightC...

Pricing Plans 1

Published pricing tiers and plan structures.

Crawlee Plans Pricing

3 plans

PLANS

Rate Limits 1

Documented rate limits and quota policies.

Crawlee Rate Limits

5 limits

RATE LIMITS

FinOps 1

Cost, billing, and metering signals for API financial operations.

Crawlee Finops

FINOPS

Security Posture 1

Authentication, domain security, vulnerability disclosure, and trust-center signals.

Crawlee Domain Security

TLSv1.3 · HSTS · DNSSEC

SECURITY

Resources

Documentation 1

Reference material describing how the API behaves

Documentation
Documentation

Build 2

SDKs, sample code, and the tooling you integrate with

GitHubOrganization
GitHubOrganization
 GitHubRepository
GitHubRepository

Access & Security 1

Authentication, authorization, and security posture

DomainSecurity
DomainSecurity

Operate 3

Status, limits, changes, and where to get help

ChangeLog
ChangeLog
 Discord
Discord
 Community
Community

Commercial 1

Pricing, plans, and the legal terms of use

License
License

Company 3

The organization behind the API

LinkedIn
LinkedIn
 Website
Website
 Blog
Blog

Other 1

Properties that don't map to a standard resource type

Apify
Apify

Source (apis.yml)

apis.yml Raw ↑ 
aid: crawlee
name: Crawlee
kind: opensource
description: Crawlee is an open-source web scraping and crawling library maintained by Apify, providing a unified set of crawler
  classes, request queues, datasets, and key-value stores for building reliable scrapers. It is available for both JavaScript/TypeScript
  (Node.js) and Python, offering HTTP, Cheerio, JSDOM, LinkeDOM, Puppeteer, Playwright, and Stagehand crawler implementations
  along with proxy and session management utilities for production-grade scraping.
url: https://raw.githubusercontent.com/api-evangelist/crawlee/refs/heads/main/apis.yml
accessModel:
  pricing: freemium
  onboarding: unknown
  trial: false
  try_now: false
  public: false
  label: Freemium
  confidence: medium
  source:
  - plans
  generated: '2026-07-22'
  method: derived
image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/icons/crawlee.png
tags:
- Apache 2.0
- Apify
- Browser Automation
- Crawlers
- Harvesting
- JavaScript
- Node.js
- Open Source
- Playwright
- Puppeteer
- Python
- Scraping
- Web
created: '2025-02-08'
modified: '2026-04-28'
specificationVersion: '0.20'
type: Index
access: Public
position: Provider
apis:
- aid: crawlee:crawlee-javascript-sdk
  name: Crawlee JavaScript SDK
  description: The Crawlee JavaScript SDK is a Node.js/TypeScript library for building reliable web scrapers and crawlers.
    It provides a family of crawler classes - BasicCrawler, HttpCrawler, CheerioCrawler, JSDOMCrawler, LinkeDOMCrawler, PuppeteerCrawler,
    PlaywrightCrawler, and AdaptivePlaywrightCrawler - along with shared infrastructure for AutoscaledPool resource management,
    proxy rotation, session pooling, RequestQueue task queuing, Dataset result storage, and KeyValueStore unstructured data
    persistence. Crawlee handles retries, error recovery, request fingerprinting, and statistics tracking out of the box,
    allowing developers to focus on extraction logic.
  humanURL: https://crawlee.dev/js
  properties:
  - type: Documentation
    url: https://crawlee.dev/js
  - type: Reference
    url: https://crawlee.dev/js/api
  - type: GettingStarted
    url: https://crawlee.dev/js/docs/quick-start
  - type: GitHubRepository
    url: https://github.com/apify/crawlee
  - type: NpmPackage
    url: https://www.npmjs.com/package/crawlee
  tags:
  - Browser Automation
  - Cheerio
  - JavaScript
  - Node.js
  - Playwright
  - Puppeteer
  - Scraping
  - TypeScript
- aid: crawlee:crawlee-python-sdk
  name: Crawlee Python SDK
  description: The Crawlee Python SDK is a Python library for building reliable web scrapers and crawlers. It offers BasicCrawler,
    HttpCrawler, BeautifulSoupCrawler, ParselCrawler, PlaywrightCrawler, and Adaptive crawlers built on top of asyncio, along
    with shared infrastructure for proxy rotation, session pooling, RequestQueue, Dataset, and KeyValueStore. The Python SDK
    targets data engineers and Python developers who want the same crawler ergonomics as the JavaScript version but inside
    the Python ecosystem.
  humanURL: https://crawlee.dev/python
  properties:
  - type: Documentation
    url: https://crawlee.dev/python
  - type: Reference
    url: https://crawlee.dev/python/api
  - type: GettingStarted
    url: https://crawlee.dev/python/docs/quick-start
  - type: GitHubRepository
    url: https://github.com/apify/crawlee-python
  - type: PyPiPackage
    url: https://pypi.org/project/crawlee/
  tags:
  - Asyncio
  - BeautifulSoup
  - Browser Automation
  - Parsel
  - Playwright
  - Python
  - Scraping
common:
- type: DomainSecurity
  url: security/crawlee-domain-security.yml
- type: LinkedIn
  url: https://www.linkedin.com/company/apify
- type: Website
  url: https://crawlee.dev/
- type: Documentation
  url: https://crawlee.dev/
- type: GitHubOrganization
  url: https://github.com/apify
- type: GitHubRepository
  url: https://github.com/apify/crawlee
- type: Blog
  url: https://crawlee.dev/blog
- type: ChangeLog
  url: https://github.com/apify/crawlee/releases
- type: Discord
  url: https://discord.gg/jyEM2PRvMU
- type: Community
  url: https://crawlee.dev/discord
- type: License
  url: https://github.com/apify/crawlee/blob/master/LICENSE.md
- type: Apify
  url: https://apify.com/
maintainers:
- FN: Kin Lane
  email: kin@apievangelist.com