WebCrawlerAPI Agent
AI-powered web crawling and scraping agent. Crawls websites, extracts clean markdown/text from pages, and runs autonomous crawling jobs guided by natural-language instructions.
AgentCard object. It is an agent card in spirit rather than in schema — useful for
discovery, but an A2A client cannot assume it will parse.
no-protocolVersionno-preferredTransporturl-is-rest-base-not-a2a-endpointnon-standard-field-mcpsecurityScheme-apiKey-in-Authorization-header
An A2A client finds this agent by fetching the well-known path on the provider's own host. That is what makes an agent card different from every other agent artifact in this catalog: it is provider-published by construction — it cannot be derived, generated, or reconstructed on a provider's behalf.
Scrape a page scrape
Fetches a single URL with a headless browser and returns its content as clean markdown, text, or HTML.
scrapemarkdownhtmljavascript-renderingCrawl a website crawl
Crawls a website starting from a seed URL, following links up to configured depth/limits, and returns extracted content for every page.
crawlwebsitemarkdownbulkAutonomous crawling agent crawling-agent
Runs an AI agent that browses a website autonomously, following a natural-language prompt to find and extract the requested information.
agentaiautonomousextraction2026-09-19 from https://thecolony.ai/.well-known/agent-card.json,
HTTP 200. The body was saved verbatim and is the sole source for
everything on this page — no field is inferred.
View the captured card.
Providers change what they serve; if this card has moved or changed shape,
the provider profile carries the
current state as of the last build.
Work with this as data
Every agent card here is available over the APIs.io API and to AI agents over MCP. A2A Agent Cards is not yet its own endpoint on the v1 API. Reach it through catalog search and the tag graph, or the MCP server.