University of Amsterdam

University of Amsterdam Robots.txt API

The Robots.txt API from University of Amsterdam — 1 operation(s) for robots.txt.

Documentation GitHub OpenAPI

Operations 1

GET /robots.txt Get Robots #

Work with this as data

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this

find_apisBrowse and filter every API in the catalog.
get_api_artifactsOne API's artifacts, grouped by type.
get_openapiThe primary OpenAPI for this API.
find_similar_apisAPIs that look like this one.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This API

curl "https://apis.io/api/v1/apis/university-of-amsterdam:university-of-amsterdam-robots-txt-api"

All apis

curl "https://apis.io/api/v1/apis?limit=25"

Calls without X-API-Key need no key. Calls with it need a paid plan — get a key.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

OpenAPI Specification

openapi: 3.2.0
info:
  title: LiteLLM Robots.txt API
  description: 'Enterprise Edition


    Proxy Server to call 100+ LLMs in the OpenAI format. **Customize Swagger Docs**


    👉 ```LiteLLM Admin Panel on /ui```. Create, Edit Keys with SSO. Having issues? Try ```Fallback Login```


    💸 ```LiteLLM Model Cost Map```.


    🔎 ```LiteLLM Model Hub```. See available models on the proxy. **Docs**'
  version: 1.95.0
tags:
- name: Robots.txt
paths:
  /robots.txt:
    get:
      summary: Get Robots
      description: 'Block all web crawlers from indexing the proxy server endpoints

        This is useful for ensuring that the API endpoints aren''t indexed by search engines'
      operationId: get_robots_robots_txt_get
      responses:
        '200':
          description: Successful Response
          content:
            application/json:
              schema: {}
      tags:
      - Robots.txt
components:
  securitySchemes:
    APIKeyHeader:
      type: apiKey
      description: Bearer token
      in: header
      name: x-litellm-api-key