niche · 0.7

Crawling

Score 17.6 / 100 9 providers 9 APIs Search apis.io →

Providers using this tag (9)

Ranked by API Evangelist rating — Exemplar and Strong are expanded by default.

Exemplar 1 Complete, well-documented, and agent-ready
Google Search ConsoleAnalyticsGoogleIndexingSearch6 APIs67.1
Strong 1 Solid coverage with minor gaps
Google IndexingGoogleIndexingSearch2 APIs65.1
Developing 2 Usable, with meaningful gaps to close
SpiderScrapingData ExtractionURLs10 APIs48.3FirecrawlData ExtractionScrapingURLs10 APIs43.1
Thin 2 Limited public surface area
ApifyActorsBrowser AutomationData Aggregation6 APIs36.4DiffbotExtractionHarvestingScrapingWeb11 APIs34.6
Emerging 3 Early or largely undocumented
Condé NastMediaPublishingMagazinesNews5 APIs15.4ZyteData ExtractionScraping1 API11.7ScrapyData ExtractionScraping1 API11.2

APIs with this tag (9)

Ranked by the provider's API Evangelist rating — the Kin Score is scored per provider, not per API, so every API of a provider shares its band. How the rating works →

Exemplar 1 Complete, well-documented, and agent-ready
Google Indexing APIThe Indexing API allows any site owner to directly notify Google when pages are added or removed, enabling ...67.1
Developing 3 Usable, with meaningful gaps to close
Spider Cloud MCP ServerSpider's hosted Model Context Protocol server exposes 22 tools — eight core operations (crawl, scrape, sear...48.3Spider Crawling APIRecursively crawl entire websites and collect every page.48.3Firecrawl Crawling APIThe Crawling API from Firecrawl — 5 operation(s) for crawling.43.1
Thin 2 Limited public surface area
Diffbot Crawl APIThe Crawl API from Diffbot — 2 operation(s) for crawl.34.6Diffbot Crawl/Bulk Job APIThe Diffbot Crawl/Bulk Job API is a powerful tool that allows users to automatically extract and organize l...34.6
Emerging 3 Early or largely undocumented
Condé Nast Brand Sitemap IndexEach brand site exposes an XML sitemap index at `/sitemap.xml` pointing to per-month sub-sitemaps (e.g. `/s...15.4ZyteZyte is a web data extraction platform providing APIs, smart proxy management, and AI-powered data extracti...11.7ScrapyScrapy is an open-source Python web crawling framework for extracting structured data from websites using s...11.2

Score breakdown

Frequency
34.4
log-scaled weighted occurrences
Breadth
0.2
spread across providers
Quality lift
44.8
mean composite of providers using it
Cohesion
0.0
strength of nearest seed neighbor

Related tags

Scraping 5 co-occurrences Data Extraction 5 co-occurrences

Where this tag comes from

Provider tag6
Api tag9
Openapi tag2
Openapi op tag7

Work with this as data

Every tag here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for tags

7 MCP tools reach this
  • find_tagsBrowse and filter every tag in the catalog.
  • get_cohortThis tag as a scored cohort — every provider carrying it, with scores.
  • cohort_statsPRO — the distribution across this tag: mean, median, band split, adoption rates.
  • cohort_rankingsPRO — the leaderboard, on composite AND agent-readiness axes.
  • apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
  • resolveTurn a domain, URL or GitHub org into the provider it belongs to.
  • find_cohortsEvery scored population of providers in the catalog.
All 92 tools

Call it yourself

curl for this page
This tag 
curl "https://apis.io/api/v1/tags/crawling"
All tags 
curl "https://apis.io/api/v1/tags?limit=25"
As a scored cohort 
curl "https://apis.io/api/v1/cohorts/tag/crawling"
The distribution (Pro) 
curl "https://apis.io/api/v1/cohorts/tag/crawling/stats" \
  -H "X-API-Key: $APIS_IO_KEY"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no email required.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.