openapi: 3.0.3
info:
title: remediation.proto Audio API
version: version not set
servers:
- url: https://api.together.xyz/v1
security:
- bearerAuth: []
tags:
- name: Audio
paths:
/audio/speech:
post:
tags:
- Audio
summary: Create audio generation request
description: Generate audio from input text
x-codeSamples:
- lang: Python
label: Together AI SDK (v2)
source: "# Docs for v1 can be found by changing the above selector ^\nfrom together import Together\nimport os\n\nclient = Together(\n api_key=os.environ.get(\"TOGETHER_API_KEY\"),\n)\n\nresponse = client.audio.speech.with_streaming_response.create(\n model=\"cartesia/sonic-2\",\n input=\"The quick brown fox jumps over the lazy dog.\",\n voice=\"laidback woman\",\n)\n\nwith response as stream:\n stream.stream_to_file(\"audio.wav\")\n"
- lang: Python
label: Together AI SDK (v1)
source: "from together import Together\nimport os\n\nclient = Together(\n api_key=os.environ.get(\"TOGETHER_API_KEY\"),\n)\n\nresponse = client.audio.speech.create(\n model=\"cartesia/sonic-2\",\n input=\"The quick brown fox jumps over the lazy dog.\",\n voice=\"laidback woman\",\n)\n\nresponse.stream_to_file(\"audio.wav\")\n"
- lang: TypeScript
label: Together AI SDK (TypeScript)
source: "import Together from \"together-ai\";\nimport { createWriteStream } from \"fs\";\nimport { join } from \"path\";\nimport { pipeline } from \"stream/promises\";\n\nconst client = new Together({\n apiKey: process.env.TOGETHER_API_KEY,\n});\n\nconst response = await client.audio.speech.create({\n model: \"cartesia/sonic-2\",\n input: \"The quick brown fox jumps over the lazy dog.\",\n voice: \"laidback woman\",\n});\n\nconst filepath = join(process.cwd(), \"audio.wav\");\nconst writeStream = createWriteStream(filepath);\n\nif (response.body) {\n await pipeline(response.body, writeStream);\n}\n"
- lang: JavaScript
label: Together AI SDK (JavaScript)
source: "import Together from \"together-ai\";\nimport { createWriteStream } from \"fs\";\nimport { join } from \"path\";\nimport { pipeline } from \"stream/promises\";\n\nconst client = new Together({\n apiKey: process.env.TOGETHER_API_KEY,\n});\n\nconst response = await client.audio.speech.create({\n model: \"cartesia/sonic-2\",\n input: \"The quick brown fox jumps over the lazy dog.\",\n voice: \"laidback woman\",\n});\n\nconst filepath = join(process.cwd(), \"audio.wav\");\nconst writeStream = createWriteStream(filepath);\n\nif (response.body) {\n await pipeline(response.body, writeStream);\n}\n"
- lang: Shell
label: cURL
source: "curl -X POST \"https://api.together.ai/v1/audio/speech\" \\\n -H \"Authorization: Bearer $TOGETHER_API_KEY\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"cartesia/sonic-2\",\n \"input\": \"The quick brown fox jumps over the lazy dog.\",\n \"voice\": \"laidback woman\"\n }' \\\n --output audio.wav\n"
operationId: audio-speech
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/AudioSpeechRequest'
responses:
'200':
description: OK
content:
application/octet-stream:
schema:
type: string
format: binary
audio/wav:
schema:
type: string
format: binary
audio/mpeg:
schema:
type: string
format: binary
text/event-stream:
schema:
$ref: '#/components/schemas/AudioSpeechStreamResponse'
'400':
description: BadRequest
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorData'
'429':
description: RateLimit
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorData'
/audio/speech/websocket:
get:
tags:
- Audio
summary: Real-time text-to-speech via WebSocket
description: "Establishes a WebSocket connection for real-time text-to-speech generation. This endpoint uses WebSocket protocol (wss://api.together.ai/v1/audio/speech/websocket) for bidirectional streaming communication.\n\n**Connection Setup:**\n- Protocol: WebSocket (wss://)\n- Authentication: Pass API key as Bearer token in Authorization header\n- Parameters: Sent as query parameters (model, voice, max_partial_length, language)\n\n**Client Events:**\n- `tts_session.updated`: Update session parameters like voice. The `session` object also accepts an `extra_params` field for additional model-specific parameters that fine-tune speech generation behavior, such as `pronunciation_dict` (a list of pronunciation rules for specific characters or symbols, where each entry uses the format `\"<source>/<replacement>\"` (e.g., `[\"omg/oh my god\"]`) to override how the model pronounces matching tokens).\n ```json\n {\n \"type\": \"tts_session.updated\",\n \"session\": {\n \"voice\": \"tara\",\n \"extra_params\": {\n \"pronunciation_dict\": [\"omg/oh my god\"]\n }\n }\n }\n ```\n- `input_text_buffer.append`: Send text chunks for TTS generation\n ```json\n {\n \"type\": \"input_text_buffer.append\",\n \"text\": \"Hello, this is a test.\"\n }\n ```\n- `input_text_buffer.clear`: Clear the buffered text\n ```json\n {\n \"type\": \"input_text_buffer.clear\"\n }\n ```\n- `input_text_buffer.commit`: Signal end of text input and process remaining text\n ```json\n {\n \"type\": \"input_text_buffer.commit\"\n }\n ```\n\n**Server Events:**\n- `session.created`: Initial session confirmation (sent first)\n ```json\n {\n \"event_id\": \"evt_123456\",\n \"type\": \"session.created\",\n \"session\": {\n \"id\": \"session-id\",\n \"object\": \"realtime.tts.session\",\n \"modalities\": [\"text\", \"audio\"],\n \"model\": \"hexgrad/Kokoro-82M\",\n \"voice\": \"tara\"\n }\n }\n ```\n- `conversation.item.input_text.received`: Acknowledgment that text was received\n ```json\n {\n \"type\": \"conversation.item.input_text.received\",\n \"text\": \"Hello, this is a test.\"\n }\n ```\n- `conversation.item.audio_output.delta`: Audio chunks as base64-encoded data\n ```json\n {\n \"type\": \"conversation.item.audio_output.delta\",\n \"item_id\": \"tts_1\",\n \"delta\": \"<base64_encoded_audio_chunk>\"\n }\n ```\n- `conversation.item.audio_output.done`: Audio generation complete for an item\n ```json\n {\n \"type\": \"conversation.item.audio_output.done\",\n \"item_id\": \"tts_1\"\n }\n ```\n- `conversation.item.tts.failed`: Error occurred\n ```json\n {\n \"type\": \"conversation.item.tts.failed\",\n \"error\": {\n \"message\": \"Error description\",\n \"type\": \"invalid_request_error\",\n \"param\": null,\n \"code\": \"invalid_api_key\"\n }\n }\n ```\n\n**Text Processing:**\n- Partial text (no sentence ending) is held in buffer until:\n - We believe that the text is complete enough to be processed for TTS generation\n - The partial text exceeds `max_partial_length` characters (default: 250)\n - The `input_text_buffer.commit` event is received\n\n**Audio Format:**\n- Format: Raw PCM (s16le, mono)\n- Sample Rate: 24000 Hz\n- Encoding: Base64 (per delta event)\n- Delivered via `conversation.item.audio_output.delta` events\n\n**Error Codes:**\n- `invalid_api_key`: Invalid API key provided (401)\n- `missing_api_key`: Authorization header missing (401)\n- `model_not_available`: Invalid or unavailable model (400)\n- Invalid text format errors (400)\n"
operationId: realtime-tts
x-codeSamples:
- lang: Python
label: Python WebSocket Client
source: "import asyncio\nimport websockets\nimport json\nimport base64\nimport os\n\nasync def generate_speech():\n api_key = os.environ.get(\"TOGETHER_API_KEY\")\n url = \"wss://api.together.ai/v1/audio/speech/websocket?model=hexgrad/Kokoro-82M&voice=af_heart\"\n\n headers = {\n \"Authorization\": f\"Bearer {api_key}\"\n }\n\n async with websockets.connect(url, additional_headers=headers) as ws:\n # Wait for session created\n session_msg = await ws.recv()\n session_data = json.loads(session_msg)\n if session_data.get(\"type\") != \"session.created\":\n print(f\"Failed to start session: {session_data}\")\n return\n print(f\"Session created: {session_data['session']['id']}\")\n\n # Send text for TTS\n text_chunks = [\n \"Hello, this is a test.\",\n \"This is the second sentence.\",\n \"And this is the final one.\"\n ]\n\n async def send_text():\n for chunk in text_chunks:\n await ws.send(json.dumps({\n \"type\": \"input_text_buffer.append\",\n \"text\": chunk\n }))\n await asyncio.sleep(0.5) # Simulate typing\n\n # Commit to process any remaining text\n await ws.send(json.dumps({\n \"type\": \"input_text_buffer.commit\"\n }))\n\n async def receive_audio():\n audio_data = bytearray()\n async for message in ws:\n data = json.loads(message)\n\n if data[\"type\"] == \"conversation.item.input_text.received\":\n print(f\"Text received: {data['text']}\")\n elif data[\"type\"] == \"conversation.item.audio_output.delta\":\n # Decode base64 audio chunk\n audio_chunk = base64.b64decode(data['delta'])\n audio_data.extend(audio_chunk)\n print(f\"Received audio chunk for item {data['item_id']}\")\n elif data[\"type\"] == \"conversation.item.audio_output.done\":\n print(f\"Audio generation complete for item {data['item_id']}\")\n elif data[\"type\"] == \"conversation.item.tts.failed\":\n error = data.get(\"error\", {})\n print(f\"Error: {error.get('message')}\")\n break\n\n # Save the raw PCM samples to a file\n with open(\"output.pcm\", \"wb\") as f:\n f.write(audio_data)\n print(\"Audio saved to output.pcm\")\n\n # Run send and receive concurrently\n await asyncio.gather(send_text(), receive_audio())\n\nasyncio.run(generate_speech())\n"
- lang: JavaScript
label: Node.js WebSocket Client
source: "import WebSocket from 'ws';\nimport fs from 'fs';\n\nconst apiKey = process.env.TOGETHER_API_KEY;\nconst url = 'wss://api.together.ai/v1/audio/speech/websocket?model=hexgrad/Kokoro-82M&voice=af_heart';\n\nconst ws = new WebSocket(url, {\n headers: {\n 'Authorization': `Bearer ${apiKey}`\n }\n});\n\nconst audioData = [];\n\nws.on('open', () => {\n console.log('WebSocket connection established!');\n});\n\nws.on('message', (data) => {\n const message = JSON.parse(data.toString());\n\n if (message.type === 'session.created') {\n console.log(`Session created: ${message.session.id}`);\n\n // Send text chunks\n const textChunks = [\n \"Hello, this is a test.\",\n \"This is the second sentence.\",\n \"And this is the final one.\"\n ];\n\n textChunks.forEach((text, index) => {\n setTimeout(() => {\n ws.send(JSON.stringify({\n type: 'input_text_buffer.append',\n text: text\n }));\n }, index * 500);\n });\n\n // Commit after all chunks\n setTimeout(() => {\n ws.send(JSON.stringify({\n type: 'input_text_buffer.commit'\n }));\n }, textChunks.length * 500 + 100);\n\n } else if (message.type === 'conversation.item.input_text.received') {\n console.log(`Text received: ${message.text}`);\n } else if (message.type === 'conversation.item.audio_output.delta') {\n // Decode base64 audio chunk\n const audioChunk = Buffer.from(message.delta, 'base64');\n audioData.push(audioChunk);\n console.log(`Received audio chunk for item ${message.item_id}`);\n } else if (message.type === 'conversation.item.audio_output.done') {\n console.log(`Audio generation complete for item ${message.item_id}`);\n } else if (message.type === 'conversation.item.tts.failed') {\n const errorMessage = message.error?.message ?? 'Unknown error';\n console.error(`Error: ${errorMessage}`);\n ws.close();\n }\n});\n\nws.on('close', () => {\n // Save the raw PCM samples to a file\n if (audioData.length > 0) {\n const completeAudio = Buffer.concat(audioData);\n fs.writeFileSync('output.pcm', completeAudio);\n console.log('Audio saved to output.pcm');\n }\n});\n\nws.on('error', (error) => {\n console.error('WebSocket error:', error);\n});\n"
parameters:
- in: query
name: model
required: false
schema:
description: The TTS model to use for speech generation. Can also be set via `tts_session.updated` event.
type: string
enum:
- hexgrad/Kokoro-82M
- cartesia/sonic-english
default: hexgrad/Kokoro-82M
- in: query
name: voice
required: false
schema:
type: string
description: 'The voice to use for speech generation. Default is ''tara''.
Available voices vary by model. Can also be updated via `tts_session.updated` event.
'
- in: query
name: max_partial_length
required: false
schema:
type: integer
default: 250
description: 'Maximum number of characters in partial text before forcing TTS generation
even without a sentence ending. Helps reduce latency for long text without punctuation.
'
- in: query
name: language
required: false
schema:
type: string
default: en
example: en
description: 'Language or locale of input text. Accepts ISO 639-1 language codes (e.g., `en`, `fr`, `es`, `zh`) as well as locale codes for region-specific variants. Locale codes must be lowercase (e.g., `zh-hk` for Cantonese). Can also be set via `tts_session.updated` event.
'
responses:
'101':
description: "Switching Protocols - WebSocket connection established successfully.\n\nError message format:\n```json\n{\n \"type\": \"conversation.item.tts.failed\",\n \"error\": {\n \"message\": \"Error description\",\n \"type\": \"invalid_request_error\",\n \"param\": null,\n \"code\": \"error_code\"\n }\n}\n```\n"
/audio/transcriptions:
post:
tags:
- Audio
summary: Create audio transcription request
description: Transcribes audio into text
x-codeSamples:
- lang: Python
label: Together AI SDK (v2)
source: "# Docs for v1 can be found by changing the above selector ^\nfrom together import Together\n\nclient = Together(\n api_key=os.environ.get(\"TOGETHER_API_KEY\"),\n)\n\nfile = open(\"audio.wav\", \"rb\")\n\nresponse = client.audio.transcriptions.create(\n model=\"openai/whisper-large-v3\",\n file=file,\n)\n\nprint(response.text)\n"
- lang: Python
label: Together AI SDK (v1)
source: "from together import Together\n\nclient = Together(\n api_key=os.environ.get(\"TOGETHER_API_KEY\"),\n)\n\nfile = open(\"audio.wav\", \"rb\")\n\nresponse = client.audio.transcriptions.create(\n model=\"openai/whisper-large-v3\",\n file=file,\n)\n\nprint(response.text)\n"
- lang: TypeScript
label: Together AI SDK (TypeScript)
source: "import Together from \"together-ai\";\nimport { readFileSync } from \"fs\";\nimport { join } from \"path\";\n\nconst client = new Together({\n apiKey: process.env.TOGETHER_API_KEY,\n});\n\nconst audioFilePath = join(process.cwd(), \"audio.wav\");\nconst audioBuffer = readFileSync(audioFilePath);\nconst audioFile = new File([audioBuffer], \"audio.wav\", { type: \"audio/wav\" });\n\nconst response = await client.audio.transcriptions.create({\n model: \"openai/whisper-large-v3\",\n file: audioFile,\n});\n\nconsole.log(response.text);\n"
- lang: JavaScript
label: Together AI SDK (JavaScript)
source: "import Together from \"together-ai\";\nimport { readFileSync } from \"fs\";\nimport { join } from \"path\";\n\nconst client = new Together({\n apiKey: process.env.TOGETHER_API_KEY,\n});\n\nconst audioFilePath = join(process.cwd(), \"audio.wav\");\nconst audioBuffer = readFileSync(audioFilePath);\nconst audioFile = new File([audioBuffer], \"audio.wav\", { type: \"audio/wav\" });\n\nconst response = await client.audio.transcriptions.create({\n model: \"openai/whisper-large-v3\",\n file: audioFile,\n});\n\nconsole.log(response.text);\n"
- lang: Shell
label: cURL
source: "curl -X POST \"https://api.together.ai/v1/audio/transcriptions\" \\\n -H \"Authorization: Bearer $TOGETHER_API_KEY\" \\\n -F \"file=@audio.wav\" \\\n -F \"model=openai/whisper-large-v3\"\n"
operationId: audio-transcriptions
requestBody:
required: true
content:
multipart/form-data:
schema:
$ref: '#/components/schemas/AudioTranscriptionRequest'
responses:
'200':
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/AudioTranscriptionResponse'
'400':
description: BadRequest
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorData'
'401':
description: Unauthorized
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorData'
'429':
description: RateLimit
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorData'
/audio/translations:
post:
tags:
- Audio
summary: Create audio translation request
description: Translates audio into English
x-codeSamples:
- lang: Python
label: Together AI SDK (v2)
source: "# Docs for v1 can be found by changing the above selector ^\nfrom together import Together\nimport os\n\nclient = Together(\n api_key=os.environ.get(\"TOGETHER_API_KEY\"),\n)\n\nfile = open(\"audio.wav\", \"rb\")\n\nresponse = client.audio.translations.create(\n model=\"openai/whisper-large-v3\",\n file=file,\n language=\"es\",\n)\n\nprint(response.text)\n"
- lang: Python
label: Together AI SDK (v1)
source: "from together import Together\nimport os\n\nclient = Together(\n api_key=os.environ.get(\"TOGETHER_API_KEY\"),\n)\n\nfile = open(\"audio.wav\", \"rb\")\n\nresponse = client.audio.translations.create(\n model=\"openai/whisper-large-v3\",\n file=file,\n language=\"es\",\n)\n\nprint(response.text)\n"
- lang: TypeScript
label: Together AI SDK (TypeScript)
source: "import Together from \"together-ai\";\nimport { readFileSync } from \"fs\";\nimport { join } from \"path\";\n\nconst client = new Together({\n apiKey: process.env.TOGETHER_API_KEY,\n});\n\nconst audioFilePath = join(process.cwd(), \"audio.wav\");\nconst audioBuffer = readFileSync(audioFilePath);\nconst audioFile = new File([audioBuffer], \"audio.wav\", { type: \"audio/wav\" });\n\nconst response = await client.audio.translations.create({\n model: \"openai/whisper-large-v3\",\n file: audioFile,\n language: \"es\"\n});\n\nconsole.log(response.text);\n"
- lang: JavaScript
label: Together AI SDK (JavaScript)
source: "import Together from \"together-ai\";\nimport { readFileSync } from \"fs\";\nimport { join } from \"path\";\n\nconst client = new Together({\n apiKey: process.env.TOGETHER_API_KEY,\n});\n\nconst audioFilePath = join(process.cwd(), \"audio.wav\");\nconst audioBuffer = readFileSync(audioFilePath);\nconst audioFile = new File([audioBuffer], \"audio.wav\", { type: \"audio/wav\" });\n\nconst response = await client.audio.translations.create({\n model: \"openai/whisper-large-v3\",\n file: audioFile,\n language: \"es\"\n});\n\nconsole.log(response.text);\n"
- lang: Shell
label: cURL
source: "curl -X POST \"https://api.together.ai/v1/audio/translations\" \\\n -H \"Authorization: Bearer $TOGETHER_API_KEY\" \\\n -F \"file=@audio.wav\" \\\n -F \"model=openai/whisper-large-v3\" \\\n -F \"language=es\"\n"
operationId: audio-translations
requestBody:
required: true
content:
multipart/form-data:
schema:
$ref: '#/components/schemas/AudioTranslationRequest'
responses:
'200':
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/AudioTranslationResponse'
'400':
description: BadRequest
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorData'
'401':
description: Unauthorized
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorData'
'429':
description: RateLimit
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorData'
/realtime:
get:
tags:
- Audio
summary: Real-time audio transcription via WebSocket
description: "Establishes a WebSocket connection for real-time audio transcription. This endpoint uses WebSocket protocol (wss://api.together.ai/v1/realtime) for bidirectional streaming communication.\n\n**Connection Setup:**\n- Protocol: WebSocket (wss://)\n- Authentication: Pass API key as Bearer token in Authorization header\n- Parameters: Sent as query parameters (model, input_audio_format)\n\n**Client Events:**\n- `input_audio_buffer.append`: Send audio chunks as base64-encoded data\n ```json\n {\n \"type\": \"input_audio_buffer.append\",\n \"audio\": \"<base64_encoded_audio_chunk>\"\n }\n ```\n- `input_audio_buffer.commit`: Signal end of audio stream. When VAD is enabled, the server automatically detects speech boundaries and emits `completed` events. When VAD is disabled, you must send `commit` to trigger transcription of the buffered audio.\n ```json\n {\n \"type\": \"input_audio_buffer.commit\"\n }\n ```\n- `transcription_session.updated`: Update session configuration, including Voice Activity Detection (VAD) parameters. Send this after receiving `session.created`. Can also be sent at any time during the session to change VAD settings.\n ```json\n {\n \"type\": \"transcription_session.updated\",\n \"session\": {\n \"turn_detection\": {\n \"type\": \"server_vad\",\n \"threshold\": 0.3,\n \"min_silence_duration_ms\": 500,\n \"min_speech_duration_ms\": 250,\n \"max_speech_duration_s\": 5.0,\n \"speech_pad_ms\": 250\n }\n }\n }\n ```\n To disable VAD entirely (manual commit mode), set `turn_detection` to `null`:\n ```json\n {\n \"type\": \"transcription_session.updated\",\n \"session\": {\n \"turn_detection\": null\n }\n }\n ```\n\n**Voice Activity Detection (VAD)**\n\nVAD controls how the server automatically detects speech segments in the audio stream. When enabled (the default), the server uses Silero VAD to identify speech regions and emits transcription events as each segment completes. When disabled, you must manually call `input_audio_buffer.commit` to trigger transcription.\n\nVAD can be configured in two ways:\n1. **Query parameters** at connection time: `turn_detection=server_vad&threshold=0.3&min_silence_duration_ms=500`\n2. **Session message** after connection: Send `transcription_session.updated` with a `turn_detection` object (see above)\n\nTo disable VAD at connection time, use `turn_detection=none` as a query parameter.\n\n**VAD Parameters:**\n\nAll parameters are optional — omitted fields use their defaults.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `type` | string | `server_vad` | VAD mode. Use `server_vad` to enable, or set `turn_detection` to `null` to disable. |\n| `threshold` | float | `0.3` | Speech probability threshold (0.0–1.0). Audio frames with probability above this value are classified as speech. Lower values detect more speech but may increase false positives. For low-SNR audio (e.g., 8kHz phone calls), values of 0.01–0.2 may work better. |\n| `min_silence_duration_ms` | int | `500` | Minimum silence duration in milliseconds before ending a speech segment. Higher values merge nearby speech bursts into single segments. For phone calls with mid-sentence pauses, 2000–5000ms prevents over-segmentation. |\n| `min_speech_duration_ms` | int | `250` | Minimum speech segment duration in milliseconds. Segments shorter than this are discarded. Filters out brief noise bursts or clicks. |\n| `max_speech_duration_s` | float | `5.0` | Maximum speech segment duration in seconds. Segments longer than this are force-split at the longest internal silence gap. Useful for continuous speech without natural pauses. |\n| `speech_pad_ms` | int | `250` | Padding in milliseconds added to the start and end of each detected segment. Prevents clipping speech edges. When padding would cause adjacent segments to overlap, the gap is split at the midpoint instead. |\n\n**Server Events:**\n- `session.created`: Initial session confirmation (sent first)\n ```json\n {\n \"type\": \"session.created\",\n \"session\": {\n \"id\": \"session-id\",\n \"object\": \"realtime.session\",\n \"modalities\": [\"audio\"],\n \"model\": \"openai/whisper-large-v3\"\n }\n }\n ```\n- `transcription_session.updated`: Confirms session configuration was applied. Sent in response to a client `transcription_session.updated` message.\n ```json\n {\n \"type\": \"transcription_session.updated\",\n \"session\": {\n \"turn_detection\": {\n \"type\": \"server_vad\",\n \"threshold\": 0.3,\n \"min_silence_duration_ms\": 500,\n \"min_speech_duration_ms\": 250,\n \"max_speech_duration_s\": 5.0,\n \"speech_pad_ms\": 250\n }\n }\n }\n ```\n- `conversation.item.input_audio_transcription.delta`: Partial transcription results\n ```json\n {\n \"type\": \"conversation.item.input_audio_transcription.delta\",\n \"delta\": \"The quick brown\"\n }\n ```\n- `conversation.item.input_audio_transcription.completed`: Final transcription\n ```json\n {\n \"type\": \"conversation.item.input_audio_transcription.completed\",\n \"transcript\": \"The quick brown fox jumps over the lazy dog\"\n }\n ```\n- `conversation.item.input_audio_transcription.failed`: Error occurred\n ```json\n {\n \"type\": \"conversation.item.input_audio_transcription.failed\",\n \"error\": {\n \"message\": \"Error description\",\n \"type\": \"invalid_request_error\",\n \"param\": null,\n \"code\": \"invalid_api_key\"\n }\n }\n ```\n\n**Error Codes:**\n- `invalid_api_key`: Invalid API key provided (401)\n- `missing_api_key`: Authorization header missing (401)\n- `model_not_available`: Invalid or unavailable model (400)\n- Unsupported audio format errors (400)\n"
operationId: realtime-transcription
x-codeSamples:
- lang: Python
label: Python WebSocket Client
source: "import asyncio\nimport websockets\nimport json\nimport base64\nimport os\n\nasync def transcribe_audio():\n api_key = os.environ.get(\"TOGETHER_API_KEY\")\n url = \"wss://api.together.ai/v1/realtime?model=openai/whisper-large-v3&input_audio_format=pcm_s16le_16000\"\n\n headers = {\n \"Authorization\": f\"Bearer {api_key}\"\n }\n\n async with websockets.connect(url, additional_headers=headers) as ws:\n # Read audio file\n with open(\"audio.wav\", \"rb\") as f:\n audio_data = f.read()\n\n # Send audio in chunks with delay to simulate real-time\n chunk_size = 8192\n bytes_per_second = 16000 * 2 # 16kHz * 2 bytes (16-bit)\n delay_per_chunk = chunk_size / bytes_per_second\n\n for i in range(0, len(audio_data), chunk_size):\n chunk = audio_data[i:i+chunk_size]\n base64_chunk = base64.b64encode(chunk).decode('utf-8')\n await ws.send(json.dumps({\n \"type\": \"input_audio_buffer.append\",\n \"audio\": base64_chunk\n }))\n # Simulate real-time streaming\n if i + chunk_size < len(audio_data):\n await asyncio.sleep(delay_per_chunk)\n\n # Commit the audio buffer\n await ws.send(json.dumps({\n \"type\": \"input_audio_buffer.commit\"\n }))\n\n # Receive transcription results\n async for message in ws:\n data = json.loads(message)\n if data[\"type\"] == \"conversation.item.input_audio_transcription.delta\":\n print(f\"Partial: {data['delta']}\")\n elif data[\"type\"] == \"conversation.item.input_audio_transcription.completed\":\n print(f\"Final: {data['transcript']}\")\n break\n elif data[\"type\"] == \"conversation.item.input_audio_transcription.failed\":\n error = data.get(\"error\", {})\n print(f\"Error: {error.get('message')}\")\n break\n\nasyncio.run(transcribe_audio())\n"
- lang: JavaScript
label: Node.js WebSocket Client
source: "import WebSocket from 'ws';\nimport fs from 'fs';\n\nconst apiKey = process.env.TOGETHER_API_KEY;\nconst url = 'wss://api.together.ai/v1/realtime?model=openai/whisper-large-v3&input_audio_format=pcm_s16le_16000';\n\nconst ws = new WebSocket(url, {\n headers: {\n 'Authorization': `Bearer ${apiKey}`\n }\n});\n\nws.on('open', async () => {\n console.log('WebSocket connection established!');\n\n // Read audio file\n const audioData = fs.readFileSync('audio.wav');\n\n // Send audio in chunks with delay to simulate real-time\n const chunkSize = 8192;\n const bytesPerSecond = 16000 * 2; // 16kHz * 2 bytes (16-bit)\n const delayPerChunk = (chunkSize / bytesPerSecond) * 1000; // Convert to ms\n\n for (let i = 0; i < audioData.length; i += chunkSize) {\n const chunk = audioData.slice(i, i + chunkSize);\n const base64Chunk = chunk.toString('base64');\n ws.send(JSON.stringify({\n type: 'input_audio_buffer.append',\n audio: base64Chunk\n }));\n\n // Simulate real-time streaming\n if (i + chunkSize < audioData.length) {\n await new Promise(resolve => setTimeout(resolve, delayPerChunk));\n }\n }\n\n // Commit audio buffer\n ws.send(JSON.stringify({\n type: 'input_audio_buffer.commit'\n }));\n});\n\nws.on('message', (data) => {\n const message = JSON.parse(data.toString());\n\n if (message.type === 'conversation.item.input_audio_transcription.delta') {\n console.log(`Partial: ${message.delta}`);\n } else if (message.type === 'conversation.item.input_audio_transcription.completed') {\n console.log(`Final: ${message.transcript}`);\n ws.close();\n } else if (message.type === 'conversation.item.input_audio_transcription.failed') {\n const errorMessage = message.error?.message ?? message.message ?? 'Unknown error';\n console.error(`Error: ${errorMessage}`);\n ws.close();\n }\n});\n\nws.on('error', (error) => {\n console.error('WebSocket error:', error);\n});\n"
parameters:
- in: query
name: model
required: true
schema:
type: string
description: The Whisper model to use for transcription
- in: query
name: input_audio_format
required: true
schema:
type: string
enum:
- pcm_s16le_16000
default: pcm_s16le_16000
description: Audio format specification. Currently supports 16-bit PCM at 16kHz sample rate.
responses:
'101':
description: "Switching Protocols - WebSocket connection established s
# --- truncated at 32 KB (48 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/together-ai/refs/heads/main/openapi/together-ai-audio-api-openapi.yml