Algolia Crawler API
Manages Algolia's hosted web crawler that extracts content from websites and pushes it into indices on a schedule. The only Algolia API that authenticates with HTTP Basic rather than the x-algolia-* header pair.
Work with this as data
Every API here is available over the APIs.io API and to AI agents over MCP.
MCP server
One button, every client — Claude, Cursor, VS Code and the rest.
https://apis.io/mcp
Tools for apis
7 MCP tools reach this
find_apisBrowse and filter every API in the catalog.get_api_artifactsOne API's artifacts, grouped by type.get_openapiThe primary OpenAPI for this API.find_similar_apisAPIs that look like this one.apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.resolveTurn a domain, URL or GitHub org into the provider it belongs to.find_cohortsEvery scored population of providers in the catalog.
Call it yourself
curl for this page
This API
curl "https://apis.io/api/v1/apis/algolia-crawler-api"
All apis
curl "https://apis.io/api/v1/apis?limit=25"Discovery needs no key. Ratings and market analysis are Pro.
Get an API key
Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.
A second provider on the same verified email joins the account you already have.
OpenAPI Specification
openapi: 3.2.0
info:
title: Crawler API
summary: The Crawler API lets you manage and run your crawlers
description: '## Base URL
The base URL for making requests to the Crawler API is:
- `https://crawler.algolia.com/api`
**All requests must use HTTPS.**
## Availability and authentication
To authenticate your API requests, use the **basic authentication** header:
- `Authorization: Basic <credentials>`
Where `<credentials>` is a base64-encoded string `<user-id>:<api-key>`.
- `<user-id>`. The Crawler user ID.
- `<api-key>`. The Crawler API key.
You can find both on the [Crawler settings](https://dashboard.algolia.com/crawler/settings) page in the Algolia dashboard.
The Crawler credentials are different from your regular Algolia credentials.
## Request format
Request bodies must be JSON objects.
## Parameters
Parameters are passed as query parameters for GET requests,
and in the request body for POST and PATCH requests.
Query parameters must be [URL-encoded](https://developer.mozilla.org/en-US/docs/Glossary/Percent-encoding).
Non-ASCII characters must be UTF-8 encoded.
## Response status and errors
The Crawler API returns JSON responses.
Since JSON doesn''t guarantee any specific ordering, don''t rely on the order of attributes in the API response.
Successful responses return a `2xx` status. Client errors return a `4xx` status.
Server errors are indicated by a `5xx` status.
Error responses have a `message` property with more information.
## Version
The current version of the Crawler API is version 1, indicated by the `/1/` in each endpoint''s URL.
'
version: 1.0.0
servers:
- url: https://crawler.algolia.com/api
description: The URL of the Crawler API.
security:
- BasicAuth: []
tags:
- name: crawler
paths:
/1/crawlers:
get:
operationId: listCrawlers
summary: List crawlers
description: Lists all your crawlers.
tags:
- crawler
x-acl:
- settings
parameters:
- $ref: '#/components/parameters/ItemsPerPage'
- $ref: '#/components/parameters/Page'
- $ref: '#/components/parameters/Name'
- $ref: '#/components/parameters/AppID'
responses:
'200':
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/crawlersResponse'
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
description: Invalid credentials.
'404':
description: The call does not have results for this combinaison of query params.
post:
operationId: createCrawler
summary: Create a crawler
description: Creates a new crawler with the provided configuration.
tags:
- crawler
x-acl:
- editSettings
requestBody:
content:
application/json:
schema:
title: crawlerCreate
type: object
properties:
name:
$ref: '#/components/schemas/CrawlerName'
config:
$ref: '#/components/schemas/Configuration'
required:
- name
- config
responses:
'200':
description: OK
content:
application/json:
schema:
title: crawlerCreateResponse
type: object
properties:
id:
$ref: '#/components/schemas/CrawlerID'
required:
- id
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
description: The user doesn't have enough rights to create a Crawler.
/1/crawlers/{id}:
get:
operationId: getCrawler
summary: Retrieve crawler details
description: 'Retrieves details about the specified crawler, optionally with its configuration.
'
tags:
- crawler
x-acl:
- settings
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
- in: query
name: withConfig
schema:
type: boolean
description: Whether the response should include the crawler's configuration.
responses:
'200':
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/GetCrawlerResponse'
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/NoRightsOnCrawler'
patch:
operationId: patchCrawler
summary: Replace crawler configuration
description: 'Replaces the crawler configuration with a new one.
If you only want to change the crawler''s name, you can use this operation.
For other configuration changes, use the "Update configuration" endpoint instead instead,
because changes made here aren''t versioned.
When replacing the configuration, you must provide the full configuration,
including any settings you want to keep.
'
externalDocs:
url: https://www.algolia.com/doc/rest-api/crawler/patch-config
description: Update configuration.
tags:
- crawler
x-acl:
- editSettings
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
requestBody:
content:
application/json:
schema:
title: patchCrawler
type: object
properties:
name:
$ref: '#/components/schemas/CrawlerName'
config:
$ref: '#/components/schemas/Configuration'
responses:
'200':
$ref: '#/components/responses/ActionAcknowledged'
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/NoRightsOnCrawler'
delete:
operationId: deleteCrawler
summary: Delete a crawler
description: Delete the specified crawler.
tags:
- crawler
x-acl:
- editSettings
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
responses:
'200':
$ref: '#/components/responses/ActionAcknowledged'
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/NoRightsOnCrawler'
/1/crawlers/{id}/run:
post:
operationId: runCrawler
summary: Unpause a crawler
description: 'Unpauses the specified crawler.
Previously ongoing crawls will be resumed.
Otherwise, the crawler waits for its next scheduled run.
'
tags:
- crawler
x-acl: []
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
responses:
'200':
$ref: '#/components/responses/ActionAcknowledged'
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/NoRightsOnCrawler'
/1/crawlers/{id}/pause:
post:
operationId: pauseCrawler
summary: Pause a crawler
description: Pauses the specified crawler.
tags:
- crawler
x-acl: []
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
responses:
'200':
$ref: '#/components/responses/ActionAcknowledged'
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/NoRightsOnCrawler'
/1/crawlers/{id}/reindex:
post:
operationId: startReindex
summary: Start a crawl
description: Starts or resumes a crawl.
tags:
- crawler
x-acl: []
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
responses:
'200':
$ref: '#/components/responses/ActionAcknowledged'
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/NoRightsOnCrawler'
/1/crawlers/{id}/test:
post:
operationId: testUrl
summary: Test crawl a URL
description: 'Tests a URL with the crawler''s configuration and shows the extracted records.
You can test configuration changes by overriding specific parts before updating the full configuration.
'
tags:
- crawler
x-acl: []
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
requestBody:
content:
application/json:
schema:
title: testUrl
type: object
properties:
url:
type: string
description: URL to test.
example: https://www.algolia.com/blog
config:
$ref: '#/components/schemas/PartialConfig'
required:
- url
responses:
'200':
description: OK
content:
application/json:
schema:
title: urlTest
type: object
properties:
startDate:
type: string
description: Date and time when the test crawl started, in RFC 3339 format.
example: '2024-04-02T15:34:29Z'
endDate:
type: string
description: Date and time when the test crawl finished, in RFC 3339 format.
example: '2024-04-02T15:34:29Z'
logs:
type: array
description: Logs from the record extraction.
items:
type: array
description: Parameters associated with each log.
items:
type: string
example: Processing url 'https://www.algolia.com/blog'
records:
type: array
description: Extracted records from the URL.
items:
title: extractedRecords
type: object
properties:
indexName:
type: string
description: Name of the index where this record will be stored.
example: testIndex
records:
type: array
description: Extracted records.
items:
type: object
description: Algolia record.
example:
objectID: https://www.algolia.com/blog
numberOfLinks: 2
recordsPerExtractor:
type: array
description: Partial records generated by each record extractor.
items:
title: recordPerExtractor
type: object
properties:
index:
type: number
description: Index of the extractor.
minimum: 0
type:
type: string
description: Extractor type.
records:
type: array
description: Extracted records.
items:
type: object
example:
- index: 0
type: custom
records:
- objectID: https://www.algolia.com/blog
links:
type: array
description: 'Links found on the page, which match the configuration and would be processed.
'
items:
type: string
example:
- https://www.algolia.com/blog/engineering/challenging-migration-heroku-google-kubernetes-engine
- https://www.algolia.com/blog/engineering/a-tale-of-two-engines-algolia-unity
externalData:
type: object
description: 'External data associated with the tested URL.
External data is refreshed automatically at the beginning of the crawl.
'
example:
externalData1:
data1: val1
data2: val2
externalData2:
data1: val1
data2: val2
error:
title: urlTestError
type: object
description: An error.
properties:
code:
type: string
example: http_internal_server_error
message:
type: string
example: HTTP Internal Server Error (500)
example: {}
required:
- startDate
- endDate
- logs
- records
- links
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/NoRightsOnCrawler'
/1/crawlers/{id}/urls/crawl:
post:
operationId: crawlUrls
summary: Crawl URLs
description: 'Crawls the specified URLs, extracts records from them, and adds them to the index.
If a crawl is running (the crawler''s `reindexing` property is `true`),
the records are added to a temporary index.
This operation is rate-limited to 500 requests every 24 hours.
'
tags:
- crawler
x-acl: []
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
requestBody:
content:
application/json:
schema:
title: crawlUrls
type: object
properties:
urls:
type: array
description: URLs to crawl.
items:
type: string
example:
- https://www.algolia.com/products/crawler/
save:
type: boolean
description: 'Whether the specified URLs should be added to the `extraURLs` property of the crawler configuration.
If unspecified, the URLs are added to the `extraUrls` field only if they haven''t been indexed during the last reindex.
'
required:
- urls
responses:
'200':
$ref: '#/components/responses/ActionAcknowledged'
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/NoRightsOnCrawler'
/1/crawlers/{id}/stats/urls:
get:
operationId: getStats
summary: Retrieve crawler stats
description: Retrieves information about the number of crawled, skipped, and failed URLs.
tags:
- crawler
x-acl:
- settings
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
responses:
'200':
description: OK
content:
application/json:
schema:
title: crawlerStats
type: object
properties:
count:
type: integer
minimum: 0
description: Total number of crawled URLs.
data:
type: array
default: []
items:
$ref: '#/components/schemas/UrlsCrawledGroup'
required:
- count
- data
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/NoRightsOnCrawler'
/1/crawlers/{id}/crawl_runs:
get:
operationId: listCrawlRuns
summary: List crawler runs
description: 'The Crawler Logs feature lets you monitor and debug your crawler’s activity by recording
detailed logs for each crawl run. Logs are useful for troubleshooting crawl issues,
verifying site coverage, and monitoring crawler performance over time.
'
tags:
- crawler
x-acl:
- settings
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
- $ref: '#/components/parameters/From'
- $ref: '#/components/parameters/Until'
- $ref: '#/components/parameters/Status'
- $ref: '#/components/parameters/Limit'
- $ref: '#/components/parameters/Offset'
- $ref: '#/components/parameters/Order'
responses:
'200':
$ref: '#/components/responses/crawlerLogsResponse'
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/NoRightsOnCrawler'
/1/crawlers/{id}/{logId}/download:
get:
operationId: getCrawlRunFile
summary: Crawler run file
description: 'The Crawler Logs file lets you monitor and debug your crawler’s activity by recording
detailed logs for each crawl run. Logs are useful for troubleshooting crawl issues,
verifying site coverage, and monitoring crawler performance over time.
'
tags:
- crawler
x-acl:
- settings
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
- $ref: '#/components/parameters/CrawlerLogIdParameter'
responses:
'200':
$ref: '#/components/responses/crawlerLogsFileResponse'
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/NoRightsOnCrawler'
/1/crawlers/{id}/delete_runs:
post:
operationId: deleteCrawlRun
summary: Crawler run record deletion
description: 'The Crawler Logs file lets you monitor and debug your crawler’s activity by recording
detailed logs for each crawl run. This delete the existing crawler log record, not the actual crawler.
'
tags:
- crawler
x-acl:
- settings
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
requestBody:
content:
application/json:
schema:
title: crawlerLogIds
type: array
items:
- $ref: '#/components/schemas/CrawlerLogID'
responses:
'200':
description: OK
content:
application/json:
schema:
title: deletedCrawlerLogIds
type: array
items:
- $ref: '#/components/schemas/CrawlerLogID'
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/NoRightsOnCrawler'
/1/crawlers/{id}/config:
patch:
operationId: patchConfig
summary: Update crawler configuration
description: 'Updates the configuration of the specified crawler.
Every time you update the configuration, a new version is created.
'
tags:
- crawler
x-acl:
- editSettings
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/PartialConfig'
responses:
'200':
$ref: '#/components/responses/ActionAcknowledged'
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/NoRightsOnCrawler'
/1/crawlers/{id}/config/versions:
get:
operationId: listConfigVersions
summary: List configuration versions
description: 'Lists previous versions of the specified crawler''s configuration, including who authored the change.
Every time you update a crawler''s configuration, a new version is added.
'
tags:
- crawler
x-acl:
- settings
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
- $ref: '#/components/parameters/ItemsPerPage'
- $ref: '#/components/parameters/Page'
responses:
'200':
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/crawlerConfigVersionsResponse'
'404':
description: Not found.
/1/crawlers/{id}/config/versions/{version}:
get:
operationId: getConfigVersion
summary: Retrieve a configuration version
description: 'Retrieves the specified version of the crawler configuration.
You can use this to restore a previous version of the configuration.
'
tags:
- crawler
x-acl:
- settings
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
- $ref: '#/components/parameters/CrawlerVersionParameter'
responses:
'200':
description: OK
content:
application/json:
schema:
title: configVersion
type: object
properties:
version:
$ref: '#/components/schemas/version'
config:
$ref: '#/components/schemas/Configuration'
createdAt:
$ref: '#/components/schemas/createdAt'
authorId:
$ref: '#/components/schemas/authorId'
required:
- version
- config
- createdAt
- authorId
'404':
description: Not found.
/1/crawlers/{id}/tasks/{taskID}:
get:
operationId: getTaskStatus
summary: Retrieve task status
description: Retrieves the status of the specified tasks, whether they're pending or completed.
tags:
- crawler
x-acl:
- settings
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
- $ref: '#/components/parameters/TaskIdParameter'
responses:
'200':
description: OK
content:
application/json:
schema:
title: taskStatus
type: object
properties:
pending:
type: boolean
description: Whether the task is pending.
required:
- pending
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/NoRightsOnCrawler'
/1/crawlers/{id}/tasks/{taskID}/cancel:
post:
operationId: cancelBlockingAction
summary: Cancel a blocking task
description: 'Cancels a blocking task.
Tasks that ran into an error will block your crawler''s schedule.
To unblock the crawler, cancel the blocking task.
'
tags:
- crawler
x-acl: []
parameters:
- $ref: '#/components/parameters/CrawlerIdParameter'
- $ref: '#/components/parameters/TaskIdParameter'
responses:
'200':
description: OK
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/NoRightsOnCrawler'
/1/domains:
get:
operationId: listDomains
summary: List registered domains
description: 'Lists registered domains.
Crawlers will only run if the URLs match any of the registered domains.
'
tags:
- crawler
x-acl:
- settings
parameters:
- $ref: '#/components/parameters/ItemsPerPage'
- $ref: '#/components/parameters/Page'
- $ref: '#/components/parameters/AppID'
responses:
'200':
description: Domains
content:
application/json:
schema:
$ref: '#/components/schemas/domainsResponse'
'400':
$ref: '#/components/responses/InvalidRequest'
'401':
$ref: '#/components/responses/MissingAuthorization'
'403':
$ref: '#/components/responses/Forbidden'
components:
schemas:
facetOrdering:
description: Order of facet names and facet values in your UI.
type: object
additionalProperties: false
properties:
facets:
$ref: '#/components/schemas/facets'
values:
$ref: '#/components/schemas/values'
crawlersResponse:
allOf:
- $ref: '#/components/schemas/Pagination'
- type: object
properties:
items:
type: array
description: Crawlers.
items:
title: crawlerResponse
type: object
properties:
id:
$ref: '#/components/schemas/CrawlerID'
name:
$ref: '#/components/schemas/CrawlerName'
loginRequestOptions:
type: object
description: Options for the HTTP request for logging in.
properties:
method:
type: string
description: HTTP method for sending the request.
default: GET
example: POST
headers:
$ref: '#/components/schemas/headers'
body:
type: string
description: Form content.
example: id=user&password=s3cr3t
timeout:
type: integer
description: Timeout for the request.
CrawlerName:
type: string
maxLength: 64
description: Name of the crawler.
example: test-crawler
widgets:
description: Widgets returned from any rules that are applied to the current search.
type: object
additionalProperties: false
properties:
banners:
$ref: '#/components/schemas/banners'
UrlsCrawledGroup:
type: object
description: Processed URLs and their status.
properties:
status:
$ref: '#/components/schemas/urlsCrawledGroupStatus'
reason:
type: string
description: Reason for this status.
category:
$ref: '#/components/schemas/urlsCrawledGroupCategory'
count:
type: integer
description: Number of URLs with this status.
readable:
type: string
description: Reason for this status.
example:
status: SKIPPED
reason: forbidden_by_robotstxt
category: fetch
count: 3
readable: Forbidden by robots.txt
cache:
type: object
description: 'Whether the crawler should cache crawled pages.
For more information, see [Partial crawls with caching](https://www.algolia.com/doc/tools/crawler/getting-started/crawler-configuration/#partial-crawls-with-caching).
'
properties:
enabled:
type: boolean
default: true
description: Whether the crawler cache is active.
maxValuesPerFacet:
type: integer
description: Maximum number of facet values to return for each facet.
default: 100
maximum: 1000
x-categories:
- Faceting
reRankingApplyFilter:
description: 'Restrict [Dynamic Re-Ranking](https://www.algolia.com/doc/guides/algolia-ai/re-ranking) to records that match these filters.
'
oneOf:
- type: array
items:
$ref: '#/components/schemas/reRankingApplyFilter'
- type: string
x-categories:
- Filtering
parameters_order:
type: string
description: 'Order of the query.
'
enum:
- ASC
- DESC
IndexSettings_advancedSyntaxFeatures:
type: array
items:
$ref: '#/components/schemas/advancedSyntaxFeatures'
description: "Advanced search syntax features you want to support\n- `exactPhrase`.\n Phrases in quotes must match exactly.\n For example, `sparkly blue \"iPhone case\"` only returns records with the exact string \"iPhone case\"\n- `excludeWords`.\n Query words prefixed with a `-` must not occur in a record.\n For example, `search -engine` matches records that contain \"search\" but not \"engine\"\nThis setting only has an effect if `advancedSyntax` is true.\n"
default:
- exactPhrase
- excludeWords
x-categories:
- Query strategy
urlsCrawledGroupStatus:
type: string
description: 'Crawled URL status.
For more information, see [Troubleshooting by crawl status](https://www.algolia.com/doc/tools/crawler/troubleshooting/crawl-status).
'
enum:
- DONE
- SKIPPED
- FAILED
ignorePlurals:
description: 'Treat singular, plurals, and other forms of declensions as equivalent.
Only use this feature for the languages used in your index.
'
example:
- ca
- es
oneOf:
- type: array
description: 'ISO code for languages for which this feature should be active.
This overrides languages you set with `queryLanguages`.
'
items:
$ref: '#/components/schemas/supportedLanguage'
- $ref: '#/components/schemas/booleanString'
- type: boolean
description: 'If true, `ignorePlurals` is active for all languages included in `queryLanguages`, or for all supported languages, if `queryLanguges` is empty.
If false, singulars, plurals, and other declensions won''t be considered equivalent.
'
default: false
x-categories:
- Languages
replaceSynonymsInHighlight:
type: boolean
description: 'Whether to replace a highlighted word with the matched synonym
By default, the original words are highlighted even if a synonym matches.
For example, with `home` as a synonym for `house` and a search for `home`,
records matching either "home" or "house" are included in the search results,
and either "home" or "house" are highlighted
With `replaceSynonymsInHighlight` set to `true`, a search for `home` still matches the same records,
but all occurrences of "house" are replaced by "home" in the highlighted response.
'
default: false
x-categories:
- Highlighting and Snippeting
restrictHighlightAndSnippetArrays:
type: boolean
description: 'Whether to restrict highlighting and snippeting to items that at least partially matched the search query.
By default, all items are highlighted and snippeted.
'
default: false
x-categories:
- Highlighting and Snippeting
disableTypoToleranceOnAttributes:
type:
# --- truncated at 32 KB (122 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/algolia/refs/heads/main/openapi/algolia-crawler-api-openapi.yml