Algolia

Algolia Crawler API

Manages Algolia's hosted web crawler that extracts content from websites and pushes it into indices on a schedule. The only Algolia API that authenticates with HTTP Basic rather than the x-algolia-* header pair.

Documentation GitHub OpenAPI

Operations 20

Each operation below carries the questions people ask an LLM about it and the instructions they give an agent to run it. Generated by API Evangelist overlay

GET /1/crawlers List crawlers · List crawlers #

Ask an LLM

“Which website crawlers do I have set up?”

“Can I find a crawler by its name?”

Tell an agent

List all my crawlers.

Find crawlers named {name}.

POST /1/crawlers Create a website crawler · Create a crawler #

Ask an LLM

“How do I set up a new crawler to index my website?”

“What configuration does a new crawler need?”

Tell an agent

Create a crawler called {name} with configuration {config}.

Set up a new site crawler {name} using this config: {config}.

GET /1/crawlers/{id} Retrieve a crawler's details · Retrieve crawler details #

Ask an LLM

“How can I check the state of one crawler, such as whether it's running or blocked?”

“Can I get a crawler's details along with its full configuration?”

Tell an agent

Get the details of crawler {id}.

Show crawler {id} including its configuration.

PATCH /1/crawlers/{id} Rename a crawler or replace its configuration · Replace crawler configuration #

Ask an LLM

“How do I rename a crawler?”

“Is there a way to overwrite a crawler's whole configuration without creating a new version?”

Tell an agent

Rename crawler {id} to {name}.

Replace crawler {id}'s entire configuration with {config}, unversioned.

DELETE /1/crawlers/{id} Delete a crawler · Delete a crawler #

Ask an LLM

“How do I permanently remove a crawler I no longer use?”

“Can I delete a website crawler entirely?”

Tell an agent destructive · confirm first

Delete crawler {id}.

Permanently remove the crawler {id}.

POST /1/crawlers/{id}/run Unpause a crawler · Unpause a crawler #

Ask an LLM

“How do I resume a crawler I paused?”

“Will unpausing a crawler continue the crawl that was in progress?”

Tell an agent

Unpause crawler {id}.

Resume the paused crawler {id} so it runs on schedule again.

POST /1/crawlers/{id}/pause Pause a crawler · Pause a crawler #

Ask an LLM

“How can I temporarily stop a crawler from running?”

“Can I put a crawler on hold while I fix my site?”

Tell an agent

Pause crawler {id}.

Put crawler {id} on hold.

POST /1/crawlers/{id}/reindex Start a full crawl · Start a crawl #

Ask an LLM

“How do I trigger a full recrawl of my website right now?”

“Can I kick off a crawl without waiting for the schedule?”

Tell an agent

Start a crawl with crawler {id}.

Reindex the whole site now using crawler {id}.

POST /1/crawlers/{id}/test Test crawl a single URL · Test crawl a URL #

Ask an LLM

“How can I preview which records a crawler would extract from a page?”

“Can I try a configuration change on one URL before saving it?”

Tell an agent

Test crawl {url} with crawler {id}.

Preview records extracted from {url} by crawler {id} using override config {config}.

POST /1/crawlers/{id}/urls/crawl Crawl specific URLs and index them · Crawl URLs #

Ask an LLM

“How do I recrawl just a handful of pages that changed?”

“What's the rate limit on crawling individual URLs?”

Tell an agent

Crawl these URLs with crawler {id}: {urls}.

Recrawl {urls} using crawler {id} and add them to its extra URLs.

GET /1/crawlers/{id}/stats/urls Get crawled, skipped and failed URL counts · Retrieve crawler stats #

Ask an LLM

“How many URLs did my crawler fail on or skip?”

“What are the URL statistics for a crawler?”

Tell an agent

Get URL stats for crawler {id}.

Show how many pages crawler {id} crawled, skipped and failed.

GET /1/crawlers/{id}/crawl_runs List a crawler's past runs · List crawler runs #

Ask an LLM

“Where can I see the history of crawl runs for troubleshooting?”

“Can I list only the failed crawl runs in a date range?”

Tell an agent

List crawl runs for crawler {id}.

Show crawler {id}'s runs with status {status} between {from} and {until}.

GET /1/crawlers/{id}/{logId}/download Download the log file for a crawl run · Crawler run file #

Ask an LLM

“How do I download the detailed log of a particular crawl run?”

“Can I get the log file to debug why pages were missed in one run?”

Tell an agent

Download log {logId} for crawler {id}.

Fetch the crawl run file {logId} from crawler {id}.

POST /1/crawlers/{id}/delete_runs Delete a crawler's run log records · Crawler run record deletion #

Ask an LLM

“How can I clear the crawl run logs without deleting the crawler?”

“Can I remove old crawl run records?”

Tell an agent destructive · confirm first

Delete the crawl run records for crawler {id}.

Clear the run logs of crawler {id} but keep the crawler.

PATCH /1/crawlers/{id}/config Update a crawler's configuration (versioned) · Update crawler configuration #

Ask an LLM

“How do I change a crawler's start URLs or schedule and keep a version history?”

“Can I update the rate limit or exclusion patterns on my crawler?”

Tell an agent

Update crawler {id}'s config with actions {actions}, app {appId} and rate limit {rateLimit}.

Set crawler {id} to start from {startUrls} on schedule {schedule}.

GET /1/crawlers/{id}/config/versions List a crawler's configuration versions · List configuration versions #

Ask an LLM

“Who changed my crawler's configuration and when?”

“Can I see previous versions of a crawler's config?”

Tell an agent

List configuration versions for crawler {id}.

Show the config change history of crawler {id}.

GET /1/crawlers/{id}/config/versions/{version} Retrieve a specific crawler configuration version · Retrieve a configuration version #

Ask an LLM

“How do I look at an older crawler configuration so I can restore it?”

“What did version 3 of my crawler config contain?”

Tell an agent

Get configuration version {version} of crawler {id}.

Show me crawler {id}'s config as of version {version}.

GET /1/crawlers/{id}/tasks/{taskID} Check a crawler task's status · Retrieve task status #

Ask an LLM

“Is my crawler task still pending or completed?”

“How do I check on a task a crawler action created?”

Tell an agent

Check the status of task {taskID} on crawler {id}.

Tell me whether crawler {id}'s task {taskID} has finished.

POST /1/crawlers/{id}/tasks/{taskID}/cancel Cancel a task that is blocking a crawler · Cancel a blocking task #

Ask an LLM

“My crawler is blocked by an errored task. How do I unblock it?”

“Can I cancel a failing task so the crawler's schedule resumes?”

Tell an agent destructive · confirm first

Cancel blocking task {taskID} on crawler {id}.

Unblock crawler {id} by cancelling its errored task {taskID}.

GET /1/domains List registered crawler domains · List registered domains #

Ask an LLM

“Which domains are registered so my crawlers are allowed to run on them?”

“Why won't my crawler run on a URL, and is its domain registered?”

Tell an agent

List my registered crawler domains.

Show the registered domains for application {appID}.

Algolia Crawler API

Operations 20

Documentation

Specifications

Schemas & Data

Other Resources

Work with this as data

MCP server

Tools for apis

Call it yourself

Get an API key

OpenAPI Specification