Mohammad Movahedi Crawlers API

The Crawlers API from Mohammad Movahedi — 3 operation(s) for crawlers.

GitHub OpenAPI

Operations 3

GET /crawlers List AI crawler registry records #
GET /crawlers/meta AI crawler registry metadata: version, counts, coverage, citation #
GET /crawlers/{token} One AI crawler registry record by user-agent token (case-insensitive) #

Work with this as data

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this
  • find_apisBrowse and filter every API in the catalog.
  • get_api_artifactsOne API's artifacts, grouped by type.
  • get_openapiThe primary OpenAPI for this API.
  • find_similar_apisAPIs that look like this one.
  • apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
  • resolveTurn a domain, URL or GitHub org into the provider it belongs to.
  • find_cohortsEvery scored population of providers in the catalog.
All 92 tools →

Call it yourself

curl for this page
This API
curl "https://apis.io/api/v1/apis/movahedi-ca:movahedi-ca-crawlers-api"
All apis
curl "https://apis.io/api/v1/apis?limit=25"

Calls without X-API-Key need no key. Calls with it need a paid plan — get a key.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. By signing up you are subscribing to the APIs.io and API Evangelist newsletters — you can unsubscribe from any issue. See our Privacy Policy and Terms.

A second provider on the same verified email joins the account you already have.

OpenAPI Specification

movahedi-ca-crawlers-api-openapi.yml Raw ↑
openapi: 3.2.0
info:
  title: movahedi.ca Public Crawlers API
  version: 1.0.0
  description: 'Free read-only API: Canadian privacy enforcement actions, a privacy glossary, Quebec Law 25 compliance guidance, a cookie and tracker pattern knowledge base, and the AI Crawler Registry. Glossary definitions support English and French via lang=en or lang=fr; other endpoint language coverage is documented per response. Anonymous quota 2,000 calls/day/IP; free API keys raise it to 10,000/day.'
  contact:
    url: https://movahedi.ca/developers
  license:
    name: Terms
    url: https://movahedi.ca/api/v1/terms
servers:
- url: https://movahedi.ca/api/v1
tags:
- name: Crawlers
paths:
  /crawlers:
    get:
      summary: List AI crawler registry records
      parameters:
      - name: q
        in: query
        schema:
          type: string
          maxLength: 128
      - name: operator
        in: query
        schema:
          type: string
          maxLength: 60
      - name: purpose
        in: query
        schema:
          type: string
          enum:
          - training
          - search-index
          - user-triggered
          - search-and-training
          - other
      - name: status
        in: query
        schema:
          type: string
          enum:
          - active
          - retired
          - renamed
      - name: limit
        in: query
        schema:
          type: integer
          minimum: 1
          maximum: 200
          default: 50
      - name: offset
        in: query
        schema:
          type: integer
          minimum: 0
          default: 0
      responses:
        '200':
          description: OK
          headers:
            X-RateLimit-Limit:
              schema:
                type: integer
            X-RateLimit-Remaining:
              schema:
                type: integer
            X-RateLimit-Reset:
              schema:
                type: integer
          content:
            application/json:
              schema:
                type: object
                required:
                - lang
                - data
                - meta
                properties:
                  lang:
                    type: string
                    enum:
                    - en
                    - fr
                    description: 'Language of the payload: the requested language, or the resolved entry language for single-term lookups.'
                  data:
                    type: object
                    required:
                    - results
                    - total_count
                    - has_more
                    - next_offset
                    properties:
                      results:
                        type: array
                        items:
                          type: object
                          required:
                          - token
                          - operator
                          - purpose
                          - honors_robots_txt
                          - vendor_doc_url
                          - first_documented
                          - status
                          - verified_at
                          properties:
                            token:
                              type: string
                            operator:
                              type: string
                            purpose:
                              type: string
                              enum:
                              - training
                              - search-index
                              - user-triggered
                              - search-and-training
                              - other
                            honors_robots_txt:
                              type: string
                              enum:
                              - 'yes'
                              - 'no'
                              - disputed
                            vendor_doc_url:
                              type: string
                              format: uri
                            ip_ranges_url:
                              type:
                              - string
                              - 'null'
                              format: uri
                            first_documented:
                              type: string
                            status:
                              type: string
                              enum:
                              - active
                              - retired
                              - renamed
                            successor:
                              type:
                              - string
                              - 'null'
                            verified_at:
                              type: string
                              format: date
                            notes:
                              type: string
                            source_note:
                              type: string
                            dataset_version:
                              type: string
                          title: Crawler
                      total_count:
                        type: integer
                      has_more:
                        type: boolean
                      next_offset:
                        type:
                        - integer
                        - 'null'
                      dataset_version:
                        type: string
                      dataset_updated:
                        type: string
                        format: date
                  meta:
                    type: object
                    required:
                    - version
                    - corpus_version
                    properties:
                      version:
                        type: string
                        description: API data bundle version, pinned to the site corpus version.
                      corpus_version:
                        type: string
                        description: movahedi.ca corpus version every tool output pins.
        '400':
          description: Invalid parameter
          content:
            application/json:
              schema:
                type: object
                required:
                - error
                properties:
                  error:
                    type: object
                    required:
                    - code
                    - message
                    - docs
                    properties:
                      code:
                        type: string
                      message:
                        type: string
                      docs:
                        type: string
                        format: uri
                      key_url:
                        type: string
                        format: uri
      tags:
      - Crawlers
      operationId: getCrawlers
      x-operation-id-source: derived
  /crawlers/meta:
    get:
      summary: 'AI crawler registry metadata: version, counts, coverage, citation'
      responses:
        '200':
          description: OK
          headers:
            X-RateLimit-Limit:
              schema:
                type: integer
            X-RateLimit-Remaining:
              schema:
                type: integer
            X-RateLimit-Reset:
              schema:
                type: integer
          content:
            application/json:
              schema:
                type: object
                required:
                - lang
                - data
                - meta
                properties:
                  lang:
                    type: string
                    enum:
                    - en
                    - fr
                    description: 'Language of the payload: the requested language, or the resolved entry language for single-term lookups.'
                  data:
                    type: object
                    required:
                    - dataset
                    - dataset_version
                    - record_count
                    properties:
                      dataset:
                        type: string
                      dataset_version:
                        type: string
                      dataset_updated:
                        type: string
                        format: date
                      record_count:
                        type: integer
                      operator_count:
                        type: integer
                      operators:
                        type: array
                        items:
                          type: string
                      purposes:
                        type: array
                        items:
                          type: string
                      by_purpose:
                        type: object
                        additionalProperties:
                          type: integer
                      statuses:
                        type: array
                        items:
                          type: string
                      last_verified:
                        type: string
                        format: date
                      citation:
                        type: string
                    title: CrawlerRegistryMeta
                  meta:
                    type: object
                    required:
                    - version
                    - corpus_version
                    properties:
                      version:
                        type: string
                        description: API data bundle version, pinned to the site corpus version.
                      corpus_version:
                        type: string
                        description: movahedi.ca corpus version every tool output pins.
      tags:
      - Crawlers
      operationId: getCrawlersMeta
      x-operation-id-source: derived
  /crawlers/{token}:
    get:
      summary: One AI crawler registry record by user-agent token (case-insensitive)
      parameters:
      - name: token
        in: path
        required: true
        schema:
          type: string
          maxLength: 80
      responses:
        '200':
          description: OK
          headers:
            X-RateLimit-Limit:
              schema:
                type: integer
            X-RateLimit-Remaining:
              schema:
                type: integer
            X-RateLimit-Reset:
              schema:
                type: integer
          content:
            application/json:
              schema:
                type: object
                required:
                - lang
                - data
                - meta
                properties:
                  lang:
                    type: string
                    enum:
                    - en
                    - fr
                    description: 'Language of the payload: the requested language, or the resolved entry language for single-term lookups.'
                  data:
                    type: object
                    required:
                    - token
                    - operator
                    - purpose
                    - honors_robots_txt
                    - vendor_doc_url
                    - first_documented
                    - status
                    - verified_at
                    properties:
                      token:
                        type: string
                      operator:
                        type: string
                      purpose:
                        type: string
                        enum:
                        - training
                        - search-index
                        - user-triggered
                        - search-and-training
                        - other
                      honors_robots_txt:
                        type: string
                        enum:
                        - 'yes'
                        - 'no'
                        - disputed
                      vendor_doc_url:
                        type: string
                        format: uri
                      ip_ranges_url:
                        type:
                        - string
                        - 'null'
                        format: uri
                      first_documented:
                        type: string
                      status:
                        type: string
                        enum:
                        - active
                        - retired
                        - renamed
                      successor:
                        type:
                        - string
                        - 'null'
                      verified_at:
                        type: string
                        format: date
                      notes:
                        type: string
                      source_note:
                        type: string
                      dataset_version:
                        type: string
                    title: Crawler
                  meta:
                    type: object
                    required:
                    - version
                    - corpus_version
                    properties:
                      version:
                        type: string
                        description: API data bundle version, pinned to the site corpus version.
                      corpus_version:
                        type: string
                        description: movahedi.ca corpus version every tool output pins.
        '404':
          description: Not found
          content:
            application/json:
              schema:
                type: object
                required:
                - error
                properties:
                  error:
                    type: object
                    required:
                    - code
                    - message
                    - docs
                    properties:
                      code:
                        type: string
                      message:
                        type: string
                      docs:
                        type: string
                        format: uri
                      key_url:
                        type: string
                        format: uri
      tags:
      - Crawlers
      operationId: getCrawlersByToken
      x-operation-id-source: derived