The Cohere Tokenize API splits input text into tokens using the tokenizer associated with a specified model. It returns both the token strings and their corresponding token IDs. This is useful for understanding how text will be processed by Cohere models, estimating token counts for billing purposes, and debugging input formatting issues.
Every API here is available over the APIs.io API and to AI agents over MCP.
One button, every client — Claude, Cursor, VS Code and the rest.
https://apis.io/mcp
find_apisBrowse and filter every API in the catalog.get_api_artifactsOne API's artifacts, grouped by type.get_openapiThe primary OpenAPI for this API.find_similar_apisAPIs that look like this one.apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.resolveTurn a domain, URL or GitHub org into the provider it belongs to.find_cohortsEvery scored population of providers in the catalog.curl "https://apis.io/api/v1/apis/cohere:tokenize-api"curl "https://apis.io/api/v1/apis?limit=25"Calls without X-API-Key need no key. Calls with it need a paid plan — get a key.
Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.
A second provider on the same verified email joins the account you already have.
openapi: 3.2.0
info:
title: Cohere Tokenize API
version: 1.0.0
servers:
- url: https://api.cohere.com
description: production
tags:
- name: Tokenize
paths:
/v1/tokenize:
post:
operationId: tokenize
summary: Tokenize
description: This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.
parameters:
- name: X-Client-Name
in: header
description: 'The name of the project that is making the request.
'
required: false
schema:
type: string
responses:
'200':
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/tokenize_Response_200'
'400':
description: "This error is returned when the request is not well formed. This could be because:\n - JSON is invalid\n - The request is missing required fields\n - The request contains an invalid combination of fields\n"
content:
application/json:
schema:
$ref: '#/components/schemas/TokenizeRequestBadRequestError'
'401':
description: "This error indicates that the operation attempted to be performed is not allowed. This could be because:\n - The api token is invalid\n - The user does not have the necessary permissions\n"
content:
application/json:
schema:
$ref: '#/components/schemas/TokenizeRequestUnauthorizedError'
'403':
description: "This error indicates that the operation attempted to be performed is not allowed. This could be because:\n - The api token is invalid\n - The user does not have the necessary permissions\n"
content:
application/json:
schema:
$ref: '#/components/schemas/TokenizeRequestForbiddenError'
'404':
description: "This error is returned when a resource is not found. This could be because:\n - The endpoint does not exist\n - The resource does not exist eg model id, dataset id\n"
content:
application/json:
schema:
$ref: '#/components/schemas/TokenizeRequestNotFoundError'
'422':
description: "This error is returned when the request is not well formed. This could be because:\n - JSON is invalid\n - The request is missing required fields\n - The request contains an invalid combination of fields\n"
content:
application/json:
schema:
$ref: '#/components/schemas/TokenizeRequestUnprocessableEntityError'
'429':
description: Too many requests
content:
application/json:
schema:
$ref: '#/components/schemas/TokenizeRequestTooManyRequestsError'
'498':
description: 'This error is returned when a request or response contains a deny-listed token.
'
content:
application/json:
schema:
$ref: '#/components/schemas/TokenizeRequestInvalidTokenError'
'499':
description: 'This error is returned when a request is cancelled by the user.
'
content:
application/json:
schema:
$ref: '#/components/schemas/TokenizeRequestClientClosedRequestError'
'500':
description: 'This error is returned when an uncategorised internal server error occurs.
'
content:
application/json:
schema:
$ref: '#/components/schemas/TokenizeRequestInternalServerError'
'501':
description: 'This error is returned when the requested feature is not implemented.
'
content:
application/json:
schema:
$ref: '#/components/schemas/TokenizeRequestNotImplementedError'
'503':
description: "This error is returned when the service is unavailable. This could be due to:\n - Too many users trying to access the service at the same time\n"
content:
application/json:
schema:
$ref: '#/components/schemas/TokenizeRequestServiceUnavailableError'
'504':
description: "This error is returned when a request to the server times out. This could be due to:\n - An internal services taking too long to respond\n"
content:
application/json:
schema:
$ref: '#/components/schemas/TokenizeRequestGatewayTimeoutError'
requestBody:
description: ''
content:
application/json:
schema:
type: object
properties:
text:
type: string
minLength: 1
maxLength: 65536
description: The string to be tokenized, the minimum text length is 1 character, and the maximum text length is 65536 characters.
model:
type: string
description: The input will be tokenized by the tokenizer that is used by this model.
required:
- text
- model
security:
- bearerAuth: []
tags:
- Tokenize
components:
schemas:
ApiMetaBilledUnits:
type: object
properties:
images:
type: number
format: double
description: 'The number of billed images.
'
input_tokens:
type: number
format: double
description: 'The number of billed input tokens.
'
image_tokens:
type: number
format: double
description: 'The number of billed image tokens.
'
output_tokens:
type: number
format: double
description: 'The number of billed output tokens.
'
search_units:
type: number
format: double
description: 'The number of billed search units.
'
classifications:
type: number
format: double
description: 'The number of billed classifications units.
'
pages:
type: number
format: double
description: 'The number of billed pages parsed.
'
title: ApiMetaBilledUnits
ApiMeta:
type: object
properties:
api_version:
$ref: '#/components/schemas/ApiMetaApiVersion'
billed_units:
$ref: '#/components/schemas/ApiMetaBilledUnits'
tokens:
$ref: '#/components/schemas/ApiMetaTokens'
cached_tokens:
type: number
format: double
description: 'The number of prompt tokens that hit the inference cache.
'
warnings:
type: array
items:
type: string
title: ApiMeta
TokenizeRequestInternalServerError:
type: object
properties:
message:
type: string
id:
type: string
title: TokenizeRequestInternalServerError
TokenizeRequestTooManyRequestsError:
type: object
properties:
message:
type: string
id:
type: string
title: TokenizeRequestTooManyRequestsError
TokenizeRequestServiceUnavailableError:
type: object
properties:
message:
type: string
id:
type: string
title: TokenizeRequestServiceUnavailableError
TokenizeRequestNotImplementedError:
type: object
properties:
message:
type: string
id:
type: string
title: TokenizeRequestNotImplementedError
TokenizeRequestInvalidTokenError:
type: object
properties:
message:
type: string
id:
type: string
title: TokenizeRequestInvalidTokenError
TokenizeRequestGatewayTimeoutError:
type: object
properties:
message:
type: string
id:
type: string
title: TokenizeRequestGatewayTimeoutError
TokenizeRequestClientClosedRequestError:
type: object
properties:
message:
type: string
id:
type: string
title: TokenizeRequestClientClosedRequestError
ApiMetaTokens:
type: object
properties:
input_tokens:
type: number
format: double
description: 'The number of tokens used as input to the model.
'
output_tokens:
type: number
format: double
description: 'The number of tokens produced by the model.
'
title: ApiMetaTokens
tokenize_Response_200:
type: object
properties:
tokens:
type: array
items:
type: integer
description: An array of tokens, where each token is an integer.
token_strings:
type: array
items:
type: string
meta:
$ref: '#/components/schemas/ApiMeta'
required:
- tokens
- token_strings
title: tokenize_Response_200
TokenizeRequestBadRequestError:
type: object
properties:
message:
type: string
id:
type: string
title: TokenizeRequestBadRequestError
ApiMetaApiVersion:
type: object
properties:
version:
type: string
is_deprecated:
type: boolean
is_experimental:
type: boolean
required:
- version
title: ApiMetaApiVersion
TokenizeRequestUnauthorizedError:
type: object
properties:
message:
type: string
id:
type: string
title: TokenizeRequestUnauthorizedError
TokenizeRequestForbiddenError:
type: object
properties:
message:
type: string
id:
type: string
title: TokenizeRequestForbiddenError
TokenizeRequestUnprocessableEntityError:
type: object
properties:
message:
type: string
id:
type: string
title: TokenizeRequestUnprocessableEntityError
TokenizeRequestNotFoundError:
type: object
properties:
message:
type: string
id:
type: string
title: TokenizeRequestNotFoundError
securitySchemes:
bearerAuth:
type: http
scheme: bearer