Scraping

An index and topic collection covering web scraping platforms, proxy networks, SERP APIs, browser-based extraction services, and data collection APIs. Scraping platforms turn the public web into structured data by combining residential and datacenter proxy networks, anti-bot circumvention, headless browser automation, and managed crawler infrastructure. This collection includes scraping APIs like ScrapingBee, Scrapfly, ScrapingAnt, ScraperAPI, and Zyte; proxy networks like Bright Data, Oxylabs, Smartproxy, SOAX, and Nimble; data extraction platforms like Apify, Diffbot, Octoparse, and Datafiniti; SERP APIs like SerpApi; AI-first crawlers like Firecrawl, Crawl4AI, Jina AI, Browser Use, and AgentQL; and open-source scraping toolkits like Scrapy, Crawlee, Beautiful Soup, and Cheerio.

Scraping is profiled on the APIs.io network. Tagged areas include Web Scraping, Data Extraction, Proxy Network, SERP API, and Residential Proxies.

The Scraping catalog on APIs.io includes 1 JSON-LD context.

Scraping’s developer surface includes developer portal and 1 more developer resources.

🌐 Visit website 📡 Profile data on GitHub

9.6/100 minimal ▬ flat Agent 0/100 human only Full breakdown ↓
scored 2026-10-23 · rubric v0.25.0 · Show this score on your site →

0 APIs 7 Features 7 Use Cases

Web Scraping Data Extraction Proxy NetworkSERP APIResidential Proxies Web CrawlingAnti-Bot CircumventionHeadless Browser

Kin Score

Kin Score How this is scored →
scored 2026-10-23 · rubric v0.25.0

Composite quality — 9.6/100 · minimal

Contract Quality 3.1 / 21

Developer Ergonomics 1.6 / 17

Access Clarity 0.0 / 17

Operational Transparency 0.6 / 11

Contract Governance 0.0 / 10

Discoverability 4.1 / 9

Regulatory Posture 0.7 / 15

Agent readiness — 0/100 · human only

Machine-Readable Contract 0 / 18

Agentic Access Contract 0 / 10

Documented Reversibility 0 / 6

MCP Server 0 / 12

Machine-Readable Auth 0 / 10

Idempotency 0 / 9

Stable Error Semantics 0 / 8

Request/Response Examples 0 / 7

Rate-Limit Signaling 0 / 7

Typed Event Surface 0 / 6

Agent Skills 0 / 5

Well-Known Catalog 0 / 4

Consent & Bot Identity 0 / 3

A2A Agent Card 0 / 8

Dry-Run / Simulate Mode 0 / 4

Delegated User Identity 0 / 6

Protected Resource Metadata 0 / 5

Registration Without a Human 0 / 6

Agentic Commerce Well-Known Document 0 / 5

Pagination 5 / 5

Distinct Operations 3 / 3

Regulatory Posture applies to this provider. Its tags matched the Horizontal (data, software, accessibility, platform) regime, so Regulatory Posture carries 15 points of the composite. If this regime is wrong for your business, say so on your provider repo — the applicability map is public and we will correct it.

Create-or-Update Ergonomics could not be measured. We hold no machine-readable contract for this provider to read, so there is nothing to measure a write surface against. Excluded rather than scored zero: never-measured and measured-empty are different facts. Publishing an OpenAPI is what makes this facet — and several others — scorable at all.

The six quality facets above are damped to 85 points between them, because the conditional facet above carries the other 15. That is why each facet's contribution is shown against a damped maximum: raising a quality facet moves the composite by 85% of its nominal weight, not 100%. The full arithmetic is at apis.io/rating/.

Show this score on your site.

It updates itself when the score changes. Paste this where you link to your API docs:

<a href="https://apis.io/providers/scraping/"><img src="https://apis.io/badge/scraping/card.svg" alt="Scraping on APIs.io — Kin Score by API Evangelist"></a>

Other styles, a README pill and a shields.io endpoint are on the badge page.

Improve this rating by publishing the missing artifacts — every area above can be raised, and the full rubric is at apis.io/rating/. Every facet and dimension name above is a link: it opens that measurement's own page — what it means, the exact checks that feed it, how the whole catalog distributes on it, and the providers at the top of it. This rating is computed from github.com/api-evangelist/scraping: open an issue to ask a question, or submit a pull request to add artifacts. Submit an artifact on GitHub — free → Manage your own listing — the Influence plan, $499/mo →

Features 7

Notable capabilities this provider offers.

Proxy Network Access

Scraping platforms expose massive pools of residential, mobile, datacenter, and ISP proxies that rotate IP addresses to distribute requests and bypass rate limits.

Anti-Bot Circumvention

Managed scraping APIs handle browser fingerprinting, TLS fingerprinting, CAPTCHA solving, and JavaScript challenges so consumers do not need to maintain their own bypass logic.

Headless Browser Rendering

Scraping APIs run real headless browsers (Chromium, Firefox, WebKit) on demand to execute JavaScript, wait for dynamic content, and capture fully rendered HTML or screenshots.

Structured Data Extraction

Platforms like Diffbot and Apify convert unstructured HTML into normalized JSON for products, articles, jobs, places, and other entity types using machine learning extraction.

SERP and Search Engine Scraping

SERP APIs like SerpApi, Bright Data SERP, and Oxylabs SERP scrape Google, Bing, Yahoo, Baidu, DuckDuckGo, and other search engines into structured JSON results.

AI-Native Web Reading

New crawlers like Firecrawl, Jina Reader, and Crawl4AI convert any URL into clean Markdown or structured JSON optimized for LLM and RAG ingestion.

Job Scheduling and Crawl Orchestration

Platforms like Apify, Octoparse, and Zyte run scheduled scraping jobs, distribute work across thousands of workers, and persist datasets for downstream consumption.

Scroll for all 7

Semantic Vocabularies 1

JSON-LD contexts and semantic vocabularies used across these APIs.

Scraping Context

7 classes · 21 properties

JSON-LD

JSON Schema 2

Standalone JSON Schema definitions for this provider's data models.

ProxyPool

12 properties

JSON SCHEMA

ScrapeJob

13 properties

JSON SCHEMA

JSON Structure 2

JSON Structure definitions describing this provider's data shapes.

Scraping Proxy Pool Structure

12 properties

JSON STRUCTURE

Scraping Scrape Job Structure

13 properties

JSON STRUCTURE

Examples 2

Example request and response payloads for these APIs.

Scraping Proxy Pool Example

12 fields

EXAMPLE

Scraping Scrape Job Example

13 fields

EXAMPLE

Use Cases 7

What developers build with this provider.

E-Commerce Price Intelligence

Retailers and marketplaces scrape competitor product pages across Amazon, Walmart, and Shopify storefronts to track pricing, availability, and assortment in near real time.

SEO and SERP Monitoring

SEO platforms use SerpApi, Bright Data, and Oxylabs SERP APIs to track keyword rankings, featured snippets, and competitor visibility across global Google locales.

Lead Generation and Sales Intelligence

Sales teams scrape LinkedIn, business directories, and review sites to enrich CRM records with contact details, company firmographics, and intent signals.

Brand and Review Monitoring

Brand teams scrape product reviews, social posts, and forums to monitor sentiment, detect counterfeits, and respond to support issues.

Real Estate and Travel Aggregation

Real estate and travel aggregators scrape listings from Zillow, Redfin, Airbnb, Booking.com, and Kayak to build search and comparison products.

AI and RAG Data Ingestion

AI teams use Firecrawl, Jina Reader, and Bright Data to crawl public web content into Markdown for retrieval-augmented generation pipelines and training datasets.

Financial and Alternative Data

Hedge funds and analysts scrape job postings, app store rankings, and pricing pages to build alternative-data signals for investment models.

Scroll for all 7

Integrations 8

Pre-built integrations with other platforms and tools.

Bright Data

Largest commercial proxy network with 150M+ residential IPs, plus managed Web Unlocker, SERP API, and Web Scraper IDE for end-to-end data collection.

Oxylabs

Premium residential, datacenter, and mobile proxies with Web Scraper API, SERP Scraper API, and E-Commerce Scraper API products.

Apify

Marketplace of 4,000+ pre-built scrapers (Actors) plus a serverless platform for running, scheduling, and storing scraped datasets.

Firecrawl

AI-native crawler that converts websites into Markdown, structured JSON, or screenshots optimized for LLM and RAG workflows.

ScrapingBee

Managed scraping API that handles headless browsers, proxy rotation, and CAPTCHA bypass with simple HTTP requests.

SerpApi

Real-time SERP scraping API supporting Google, Bing, Yahoo, Baidu, YouTube, Amazon, eBay, and 30+ other search engines with structured JSON output.

Diffbot

AI-powered structured extraction across articles, products, discussions, videos, and a public Knowledge Graph of 10B+ entities.

Zyte

End-to-end scraping platform from the creators of Scrapy, with Smart Proxy Manager, automatic unblocking, and structured data APIs.

Scroll for all 8

Resources

Get Started 1

Portal, sign-up, and the first successful call

Portal

Portal

Build 1

SDKs, sample code, and the tooling you integrate with

GitHubOrganization

GitHubOrganization

Source (apis.yml)

aid: scraping
name: Scraping
description: An index and topic collection covering web scraping platforms, proxy networks, SERP APIs, browser-based extraction
  services, and data collection APIs. Scraping platforms turn the public web into structured data by combining residential
  and datacenter proxy networks, anti-bot circumvention, headless browser automation, and managed crawler infrastructure.
  This collection includes scraping APIs like ScrapingBee, Scrapfly, ScrapingAnt, ScraperAPI, and Zyte; proxy networks like
  Bright Data, Oxylabs, Smartproxy, SOAX, and Nimble; data extraction platforms like Apify, Diffbot, Octoparse,
  and Datafiniti; SERP APIs like SerpApi; AI-first crawlers like Firecrawl, Crawl4AI, Jina AI, Browser Use, and AgentQL; and
  open-source scraping toolkits like Scrapy, Crawlee, Beautiful Soup, and Cheerio.
type: Index
image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/apis-json-logo.jpg
tags:
- Web Scraping
- Data Extraction
- Proxy Network
- SERP API
- Residential Proxies
- Web Crawling
- Anti-Bot Circumvention
- Headless Browser
url: https://raw.githubusercontent.com/api-evangelist/scraping/refs/heads/main/apis.yml
created: '2026-05-19'
modified: '2026-05-19'
specificationVersion: '0.19'
apis: []
network:
- name: AgentQL
  url: https://raw.githubusercontent.com/api-evangelist/agentql/refs/heads/main/apis.yml
- name: Apify
  url: https://raw.githubusercontent.com/api-evangelist/apify/refs/heads/main/apis.yml
- name: Beautiful Soup
  url: https://raw.githubusercontent.com/api-evangelist/beautiful-soup/refs/heads/main/apis.yml
- name: Bright Data
  url: https://raw.githubusercontent.com/api-evangelist/bright-data/refs/heads/main/apis.yml
- name: Browser Use
  url: https://raw.githubusercontent.com/api-evangelist/browser-use/refs/heads/main/apis.yml
- name: Cheerio
  url: https://raw.githubusercontent.com/api-evangelist/cheerio/refs/heads/main/apis.yml
- name: Crawl4AI
  url: https://raw.githubusercontent.com/api-evangelist/crawl4ai/refs/heads/main/apis.yml
- name: Crawlee
  url: https://raw.githubusercontent.com/api-evangelist/crawlee/refs/heads/main/apis.yml
- name: Datafiniti
  url: https://raw.githubusercontent.com/api-evangelist/datafiniti/refs/heads/main/apis.yml
- name: Diffbot
  url: https://raw.githubusercontent.com/api-evangelist/diffbot/refs/heads/main/apis.yml
- name: Firecrawl
  url: https://raw.githubusercontent.com/api-evangelist/firecrawl/refs/heads/main/apis.yml
- name: Foodspark
  url: https://raw.githubusercontent.com/api-evangelist/foodspark/refs/heads/main/apis.yml
- name: Import.io
  url: https://raw.githubusercontent.com/api-evangelist/import-io/refs/heads/main/apis.yml
- name: Jina AI
  url: https://raw.githubusercontent.com/api-evangelist/jina-ai/refs/heads/main/apis.yml
- name: Nimble
  url: https://raw.githubusercontent.com/api-evangelist/nimble/refs/heads/main/apis.yml
- name: Octoparse
  url: https://raw.githubusercontent.com/api-evangelist/octoparse/refs/heads/main/apis.yml
- name: Oxylabs
  url: https://raw.githubusercontent.com/api-evangelist/oxylabs/refs/heads/main/apis.yml
- name: ParseHub
  url: https://raw.githubusercontent.com/api-evangelist/parsehub/refs/heads/main/apis.yml
- name: ScraperAPI
  url: https://raw.githubusercontent.com/api-evangelist/scraper-api/refs/heads/main/apis.yml
- name: Scrapfly
  url: https://raw.githubusercontent.com/api-evangelist/scrapfly/refs/heads/main/apis.yml
- name: ScrapingAnt
  url: https://raw.githubusercontent.com/api-evangelist/scrapingant/refs/heads/main/apis.yml
- name: ScrapingBee
  url: https://raw.githubusercontent.com/api-evangelist/scrapingbee/refs/heads/main/apis.yml
- name: Scrapy
  url: https://raw.githubusercontent.com/api-evangelist/scrapy/refs/heads/main/apis.yml
- name: SerpApi
  url: https://raw.githubusercontent.com/api-evangelist/serpapi/refs/heads/main/apis.yml
- name: Smartproxy
  url: https://raw.githubusercontent.com/api-evangelist/smartproxy/refs/heads/main/apis.yml
- name: SOAX
  url: https://raw.githubusercontent.com/api-evangelist/soax/refs/heads/main/apis.yml
- name: Zyte
  url: https://raw.githubusercontent.com/api-evangelist/zyte/refs/heads/main/apis.yml
common:
- type: Portal
  url: https://apievangelist.com
- type: GitHubOrganization
  url: https://github.com/api-evangelist
- type: Features
  data:
  - name: Proxy Network Access
    description: Scraping platforms expose massive pools of residential, mobile, datacenter, and ISP proxies that rotate IP
      addresses to distribute requests and bypass rate limits.
  - name: Anti-Bot Circumvention
    description: Managed scraping APIs handle browser fingerprinting, TLS fingerprinting, CAPTCHA solving, and JavaScript
      challenges so consumers do not need to maintain their own bypass logic.
  - name: Headless Browser Rendering
    description: Scraping APIs run real headless browsers (Chromium, Firefox, WebKit) on demand to execute JavaScript, wait
      for dynamic content, and capture fully rendered HTML or screenshots.
  - name: Structured Data Extraction
    description: Platforms like Diffbot and Apify convert unstructured HTML into normalized JSON for products, articles, jobs,
      places, and other entity types using machine learning extraction.
  - name: SERP and Search Engine Scraping
    description: SERP APIs like SerpApi, Bright Data SERP, and Oxylabs SERP scrape Google, Bing, Yahoo, Baidu, DuckDuckGo,
      and other search engines into structured JSON results.
  - name: AI-Native Web Reading
    description: New crawlers like Firecrawl, Jina Reader, and Crawl4AI convert any URL into clean Markdown or structured
      JSON optimized for LLM and RAG ingestion.
  - name: Job Scheduling and Crawl Orchestration
    description: Platforms like Apify, Octoparse, and Zyte run scheduled scraping jobs, distribute work across thousands of
      workers, and persist datasets for downstream consumption.
- type: UseCases
  data:
  - name: E-Commerce Price Intelligence
    description: Retailers and marketplaces scrape competitor product pages across Amazon, Walmart, and Shopify storefronts
      to track pricing, availability, and assortment in near real time.
  - name: SEO and SERP Monitoring
    description: SEO platforms use SerpApi, Bright Data, and Oxylabs SERP APIs to track keyword rankings, featured snippets,
      and competitor visibility across global Google locales.
  - name: Lead Generation and Sales Intelligence
    description: Sales teams scrape LinkedIn, business directories, and review sites to enrich CRM records with contact details,
      company firmographics, and intent signals.
  - name: Brand and Review Monitoring
    description: Brand teams scrape product reviews, social posts, and forums to monitor sentiment, detect counterfeits, and
      respond to support issues.
  - name: Real Estate and Travel Aggregation
    description: Real estate and travel aggregators scrape listings from Zillow, Redfin, Airbnb, Booking.com, and Kayak to
      build search and comparison products.
  - name: AI and RAG Data Ingestion
    description: AI teams use Firecrawl, Jina Reader, and Bright Data to crawl public web content into Markdown for retrieval-augmented
      generation pipelines and training datasets.
  - name: Financial and Alternative Data
    description: Hedge funds and analysts scrape job postings, app store rankings, and pricing pages to build alternative-data
      signals for investment models.
- type: Integrations
  data:
  - name: Bright Data
    description: Largest commercial proxy network with 150M+ residential IPs, plus managed Web Unlocker, SERP API, and Web
      Scraper IDE for end-to-end data collection.
  - name: Oxylabs
    description: Premium residential, datacenter, and mobile proxies with Web Scraper API, SERP Scraper API, and E-Commerce
      Scraper API products.
  - name: Apify
    description: Marketplace of 4,000+ pre-built scrapers (Actors) plus a serverless platform for running, scheduling, and
      storing scraped datasets.
  - name: Firecrawl
    description: AI-native crawler that converts websites into Markdown, structured JSON, or screenshots optimized for LLM
      and RAG workflows.
  - name: ScrapingBee
    description: Managed scraping API that handles headless browsers, proxy rotation, and CAPTCHA bypass with simple HTTP
      requests.
  - name: SerpApi
    description: Real-time SERP scraping API supporting Google, Bing, Yahoo, Baidu, YouTube, Amazon, eBay, and 30+ other search
      engines with structured JSON output.
  - name: Diffbot
    description: AI-powered structured extraction across articles, products, discussions, videos, and a public Knowledge Graph
      of 10B+ entities.
  - name: Zyte
    description: End-to-end scraping platform from the creators of Scrapy, with Smart Proxy Manager, automatic unblocking,
      and structured data APIs.
maintainers:
- FN: Kin Lane
  email: kin@apievangelist.com
humanURL: https://apievangelist.com

Work with this as data

Every provider here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for providers

9 MCP tools reach this

find_providersBrowse and filter every provider in the catalog.
get_provider_artifactsEvery artifact this provider publishes, grouped by type.
get_provider_operationsEvery operation across all of their OpenAPIs — one call instead of parsing every spec.
get_provider_toolsEvery MCP tool they ship, with the operation each wraps.
get_provider_evidenceHow each part of their score was established. Free — the basis for a claim should not sit behind it.
get_provider_ratingPRO — composite, band, trend and facet scores.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This provider

curl "https://apis.io/api/v1/providers/scraping"

All providers

curl "https://apis.io/api/v1/providers?limit=25"

Every operation they expose

curl "https://apis.io/api/v1/providers/scraping/operations?limit=25"

How their score was established

curl "https://apis.io/api/v1/providers/scraping/evidence"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.