AI Crawler Index

AI Crawler Index Ip Ranges API

Operator-published prefixes, normalised.

Documentation GitHub OpenAPI

Operations 4

GET /data/ip-sources.json Which operators publish verifiable IP ranges, and where #

GET /ip-ranges/all.json Union of every operator-published prefix, grouped by source #

GET /ip-ranges/all.txt The same prefixes, one CIDR per line #

GET /ip-ranges/{source}.json One operator's published prefix list, normalised #

Work with this as data

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this

find_apisBrowse and filter every API in the catalog.
get_api_artifactsOne API's artifacts, grouped by type.
get_openapiThe primary OpenAPI for this API.
find_similar_apisAPIs that look like this one.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This API

curl "https://apis.io/api/v1/apis/pathwren:pathwren-ip-ranges-api"

All apis

curl "https://apis.io/api/v1/apis?limit=25"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

OpenAPI Specification

openapi: 3.2.0
info:
  title: AI Crawler Index Ip Ranges API
  version: '2026-09-01'
  summary: Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex.
  description: 'A read-only, static, keyless index of 56 web crawlers operated by 30 companies and projects: what each one is for, the exact robots.txt token and user-agent substring, whether the operator says it obeys robots.txt, how to verify it is genuine, and — the part nobody else publishes — what you lose by blocking it.'
  license:
    name: CC0-1.0
    url: https://creativecommons.org/publicdomain/zero/1.0/
  contact:
    url: https://www.pathwren.workers.dev/about.html
servers:
- url: https://www.pathwren.workers.dev
tags:
- name: IP Ranges
  description: Operator-published prefixes, normalised.
paths:
  /data/ip-sources.json:
    get:
      summary: Which operators publish verifiable IP ranges, and where
      operationId: listIpSources
      tags:
      - IP Ranges
      responses:
        '200':
          description: Source list
          content:
            application/json: {}
  /ip-ranges/all.json:
    get:
      summary: Union of every operator-published prefix, grouped by source
      operationId: getAllIpRanges
      tags:
      - IP Ranges
      responses:
        '200':
          description: 1887 IPv4 and 1056 IPv6 prefixes
          content:
            application/json: {}
  /ip-ranges/all.txt:
    get:
      summary: The same prefixes, one CIDR per line
      tags:
      - IP Ranges
      operationId: getAllIpRangesText
      responses:
        '200':
          description: 'Plain text, # comments'
          content:
            text/plain:
              schema:
                type: string
  /ip-ranges/{source}.json:
    get:
      summary: One operator's published prefix list, normalised
      operationId: getIpRangeSource
      tags:
      - IP Ranges
      parameters:
      - name: source
        in: path
        required: true
        description: Source slug, from /data/ip-sources.json
        schema:
          type: string
          enum:
          - openai-gptbot
          - openai-searchbot
          - openai-chatgpt-user
          - google-googlebot
          - google-special
          - google-user-triggered
          - google-user-triggered-google
          - bing-bingbot
          - apple-applebot
          - duckduckgo-duckduckbot
          - perplexity-bot
          - perplexity-user
      responses:
        '200':
          description: Prefixes with provenance
          content:
            application/json: {}
        '404':
          description: No such source

AI Crawler Index Ip Ranges API

Operations 4

Documentation

Specifications

Schemas & Data

Other Resources