velma2PiiPhiRedactionStreaming
Establish a WebSocket connection for real-time PII/PHI redaction. The service transcribes audio in real-time, detects PII/PHI in the text, redacts the transcript, and returns redacted audio clips as binary MP3 frames. ## Connection Flow 1. Connect with `api_key` and optional query parameters 2. Connection will be accepted if authentication succeeds and limits are not exceeded 3. Stream audio data as binary WebSocket frames 4. Receive utterance results as JSON+binary frame pairs (redacted text + MP3 clip) 5. Send empty text frame ("") when audio stream ends 6. Receive final "done" message with optional trailing audio 7. Connection closes automatically after completion ## Error Handling If an error occurs during processing, the server sends an error JSON message and closes the connection. Connection rejections during handshake use WebSocket close codes. ## WebSocket Close Codes - `1000`: Normal closure after the `done` message, or after the connection completes. - `1003`: Invalid connection parameter (e.g. unsupported `audio_format`, `sample_rate`, `num_channels`, an invalid `language`, or a raw format missing `sample_rate`/`num_channels`). An `error` message is sent before the close. - `4003`: The request could not be validated, or the request is not permitted. An `error` message is sent before the close. - `4029`: The request could not be completed due to insufficient credits. An `error` message is sent before the close.
asyncapi: 3.0.0
info:
title: Velma 2 PII/PHI Redaction Streaming API
version: 0.0.0
description: >
Real-time speech-to-text with PII/PHI redaction over WebSocket. Provides
live
transcription with automatic language detection, PII/PHI detection and text
redaction, and a redacted MP3 audio clip alongside each utterance with the
PII/PHI ranges silenced. Optional speaker diarization and configurable
redaction padding are selected via query parameters.
servers:
production:
host: platform.modulate.ai
protocol: wss
description: Modulate PII/PHI Redaction Streaming Server
channels:
velma2PiiPhiRedactionStreaming:
address: /api/velma-2-pii-phi-redaction-streaming
description: >
Establish a WebSocket connection for real-time PII/PHI redaction. The
service
transcribes audio in real-time, detects PII/PHI in the text, redacts the
transcript, and returns redacted audio clips as binary MP3 frames.
## Connection Flow
1. Connect with `api_key` and optional query parameters
2. Connection will be accepted if authentication succeeds and limits are
not exceeded
3. Stream audio data as binary WebSocket frames
4. Receive utterance results as JSON+binary frame pairs (redacted text +
MP3 clip)
5. Send empty text frame ("") when audio stream ends
6. Receive final "done" message with optional trailing audio
7. Connection closes automatically after completion
## Error Handling
If an error occurs during processing, the server sends an error JSON
message and
closes the connection. Connection rejections during handshake use
WebSocket close codes.
## WebSocket Close Codes
- `1000`: Normal closure after the `done` message, or after the
connection completes.
- `1003`: Invalid connection parameter (e.g. unsupported `audio_format`,
`sample_rate`,
`num_channels`, an invalid `language`, or a raw format missing
`sample_rate`/`num_channels`). An `error` message is sent before the close.
- `4003`: The request could not be validated, or the request is not
permitted.
An `error` message is sent before the close.
- `4029`: The request could not be completed due to insufficient credits.
An `error` message is sent before the close.
bindings:
ws:
query:
type: object
required:
- api_key
properties:
api_key:
type: string
description: >
Your API key for authentication. Required. Must be provided as a
query
parameter when establishing the WebSocket connection; the
connection is
rejected during the handshake if it is missing.
speaker_diarization:
type: boolean
default: true
description: >
Enable speaker diarization to identify different speakers in the
audio stream.
When enabled, each utterance includes a speaker identifier
(1-indexed integer).
audio_format:
type: string
description: >
Audio encoding format. Determines how the service interprets
incoming audio data.
**Self-describing formats** (`wav`, `mp3`, `ogg`, `flac`,
`webm`, `aac`, `aiff`):
These include container formats (AIFF, OGG, WAV, WebM) and
codecs whose streams
carry per-frame headers (AAC/ADTS, FLAC, MP3). When omitted
(default), the service
auto-detects the format from these headers. You may specify the
format explicitly
to override auto-detection. Do not specify `sample_rate` or
`num_channels` for
self-describing formats - the headers already carry this
metadata.
**Raw/headerless formats** (`s8`..`f64be`, `mulaw`, `alaw`):
These formats are raw sample data with no headers. When using a
raw format, you
must also specify `sample_rate` and `num_channels` so the
service knows how to
interpret the byte stream. Omitting either will result in an
error.
enum:
- s8
- s16le
- s16be
- s24le
- s24be
- s32le
- s32be
- u8
- u16le
- u16be
- u24le
- u24be
- u32le
- u32be
- f32le
- f32be
- f64le
- f64be
- mulaw
- alaw
- wav
- mp3
- ogg
- flac
- webm
- aac
- aiff
sample_rate:
type: integer
description: >
Audio sample rate in Hz. Required when `audio_format` is a
raw/headerless format
(PCM, mu-law, A-law). Do not specify for self-describing formats
(WAV, MP3, OGG,
etc.) - their headers already include this metadata. Cannot be
specified without
`audio_format`.
enum:
- 8000
- 11025
- 16000
- 22050
- 32000
- 44100
- 48000
- 96000
num_channels:
type: integer
minimum: 1
maximum: 8
description: >
Number of audio channels (1-8). Required when `audio_format` is
a raw/headerless
format (PCM, mu-law, A-law). Do not specify for self-describing
formats (WAV, MP3,
OGG, etc.) - their headers already include this metadata. Use 1
for mono, 2 for
stereo. Cannot be specified without `audio_format`.
start_redaction_padding_ms:
type: integer
minimum: 0
default: 100
description: >
Additional silence in milliseconds to prepend before each
redacted audio range.
Extends the silenced region earlier in time to provide a buffer
before PII/PHI content.
end_redaction_padding_ms:
type: integer
minimum: 0
default: 0
description: >
Additional silence in milliseconds to append after each redacted
audio range.
Extends the silenced region later in time to provide a buffer
after PII/PHI content.
language:
type: string
pattern: ^[A-Za-z]{2,3}([-_][A-Za-z0-9]+)*$
description: >
Optional language hint as a case-insensitive ISO 639-1 code
(e.g. "en",
"fr"). BCP 47 region/script subtags (e.g. "en-US") are accepted;
only the
primary language subtag is used. When omitted, the language is
detected
automatically for each utterance.
messages:
audioChunk:
$ref: '#/components/messages/AudioChunk'
endOfStream:
$ref: '#/components/messages/EndOfStream'
utteranceResult:
$ref: '#/components/messages/UtteranceResult'
redactedAudioClip:
$ref: '#/components/messages/RedactedAudioClip'
errorResult:
$ref: '#/components/messages/ErrorResponse'
doneResult:
$ref: '#/components/messages/DoneResult'
trailingAudioClip:
$ref: '#/components/messages/TrailingAudioClip'
operations:
streamAudio:
action: send
channel:
$ref: '#/channels/velma2PiiPhiRedactionStreaming'
summary: Stream audio for real-time PII/PHI redaction
messages:
- $ref: '#/channels/velma2PiiPhiRedactionStreaming/messages/audioChunk'
- $ref: '#/channels/velma2PiiPhiRedactionStreaming/messages/endOfStream'
receiveRedaction:
action: receive
channel:
$ref: '#/channels/velma2PiiPhiRedactionStreaming'
summary: Receive redacted transcription results and audio clips from the server
messages:
- $ref: '#/channels/velma2PiiPhiRedactionStreaming/messages/utteranceResult'
- $ref: '#/channels/velma2PiiPhiRedactionStreaming/messages/redactedAudioClip'
- $ref: '#/channels/velma2PiiPhiRedactionStreaming/messages/errorResult'
- $ref: '#/channels/velma2PiiPhiRedactionStreaming/messages/doneResult'
- $ref: '#/channels/velma2PiiPhiRedactionStreaming/messages/trailingAudioClip'
components:
messages:
AudioChunk:
name: audioChunk
title: Audio Data Chunk
summary: Audio data chunk for real-time redaction
contentType: application/octet-stream
payload:
type: string
format: binary
EndOfStream:
name: endOfStream
title: End of Audio Stream
summary: Empty text frame signaling end of audio stream
contentType: text/plain
payload:
type: string
const: ''
examples:
- name: endSignal
payload: ''
UtteranceResult:
name: utteranceResult
title: Utterance Result
summary: >
A completed utterance with redacted text. When `redacted_audio` is not
null,
the next frame from the server is a binary MP3 frame containing the
redacted
audio clip. When `redacted_audio` is null (out-of-order utterance whose
audio
was already emitted), no binary frame follows.
contentType: application/json
payload:
type: object
required:
- type
- utterance
- redacted_audio
properties:
type:
type: string
const: utterance
utterance:
$ref: '#/components/schemas/StreamingUtterance'
redacted_audio:
oneOf:
- $ref: '#/components/schemas/RedactedAudioInfo'
- type: 'null'
description: >
Describes the audio clip that follows as the next binary frame.
`start_ms` and `duration_ms` indicate the time window in the
original
audio stream that the clip covers (from the last emitted point to
the
end of this utterance). Null when the utterance's audio was
already
emitted in a previous window (out-of-order); in that case, no
binary
frame follows.
examples:
- name: utteranceWithAudio
payload:
type: utterance
utterance:
utterance_uuid: b2c3d4e5-f6a7-8901-bcde-f12345678901
text: Hello, my name is <pii:name></pii:name>.
start_ms: 0
duration_ms: 3000
speaker: 1
language: en
redacted_audio:
start_ms: 0
duration_ms: 3000
- name: outOfOrderUtterance
summary: >-
Out-of-order utterance (audio already emitted, no binary frame
follows)
payload:
type: utterance
utterance:
utterance_uuid: c3d4e5f6-a7b8-9012-cdef-234567890123
text: Yes.
start_ms: 500
duration_ms: 400
speaker: 2
language: en
redacted_audio: null
RedactedAudioClip:
name: redactedAudioClip
title: Redacted Audio Clip
summary: |
Binary MP3 data for the redacted audio window. Sent immediately after an
utterance JSON frame that has a non-null `redacted_audio` field.
contentType: audio/mpeg
payload:
type: string
format: binary
ErrorResponse:
name: errorResult
title: Error
summary: An error occurred during redaction. No binary frame follows.
contentType: application/json
payload:
type: object
required:
- type
- error
properties:
type:
type: string
const: error
error:
type: string
description: Human-readable error message
examples:
- name: internalError
payload:
type: error
error: Internal server error
- name: invalidInputAudio
payload:
type: error
error: Invalid input audio
- name: notPermitted
payload:
type: error
error: This request is not permitted.
- name: insufficientCredits
payload:
type: error
error: Insufficient credits.
- name: validationUnavailable
payload:
type: error
error: Unable to validate request. Please try again.
- name: invalidAudioFormat
payload:
type: error
error: >-
Invalid audio_format='mp4'. Valid values: ['aac', 'aiff', 'alaw',
'f32be', 'f32le', 'f64be', 'f64le', 'flac', 'mp3', 'mulaw', 'ogg',
's16be', 's16le', 's24be', 's24le', 's32be', 's32le', 's8',
'u16be', 'u16le', 'u24be', 'u24le', 'u32be', 'u32le', 'u8', 'wav',
'webm']
- name: rawFormatMissingParams
payload:
type: error
error: >-
Raw audio formats require sample_rate and num_channels query
parameters
- name: paramsWithoutFormat
payload:
type: error
error: sample_rate and num_channels require audio_format to be specified
- name: invalidSampleRate
payload:
type: error
error: >-
Invalid sample_rate=50000. Valid values: [8000, 11025, 16000,
22050, 32000, 44100, 48000, 96000]
- name: invalidNumChannels
payload:
type: error
error: Invalid num_channels=12. Must be between 1 and 8
DoneResult:
name: doneResult
title: Redaction Complete
summary: >
Redaction completed successfully. When `trailing_redacted_audio` is not
null,
the next frame is a binary MP3 frame containing the remaining audio
after the
last utterance. When null, no binary frame follows.
contentType: application/json
payload:
type: object
required:
- type
- duration_ms
- trailing_redacted_audio
properties:
type:
type: string
const: done
duration_ms:
type: integer
minimum: 0
description: Total duration of audio processed in milliseconds
trailing_redacted_audio:
oneOf:
- $ref: '#/components/schemas/RedactedAudioInfo'
- type: 'null'
description: >
Describes any remaining audio after the last utterance. When not
null,
a binary MP3 frame follows containing the trailing audio with any
applicable redaction applied. Null when the last utterance already
covered the end of the audio.
examples:
- name: completionWithTrailingAudio
payload:
type: done
duration_ms: 45000
trailing_redacted_audio:
start_ms: 43000
duration_ms: 2000
- name: completionNoTrailingAudio
payload:
type: done
duration_ms: 45000
trailing_redacted_audio: null
TrailingAudioClip:
name: trailingAudioClip
title: Trailing Audio Clip
summary: |
Binary MP3 data for the trailing audio after the last utterance. Sent
immediately after a `done` JSON frame that has a non-null
`trailing_redacted_audio` field.
contentType: audio/mpeg
payload:
type: string
format: binary
schemas:
StreamingUtterance:
type: object
required:
- utterance_uuid
- text
- start_ms
- duration_ms
- speaker
- language
properties:
utterance_uuid:
type: string
format: uuid
description: Unique identifier for this utterance.
example: a1b2c3d4-e5f6-7890-abcd-ef1234567890
text:
type: string
description: >
The redacted text for this utterance. Each detected PII or PHI span
is
replaced with an empty marker tag - `<phi></phi>` for health
information
and `<pii:CATEGORY></pii:CATEGORY>` for personal information, where
CATEGORY identifies the detected entity type - with the surrounding
text
preserved.
example: Hello, my name is <pii:name></pii:name>.
start_ms:
type: integer
minimum: 0
description: >
The start time of this utterance in milliseconds, relative to the
beginning
of the audio stream.
example: 0
duration_ms:
type: integer
minimum: 0
description: The duration of this utterance in milliseconds.
example: 2500
speaker:
type: integer
description: >
Speaker identifier for this utterance (1-indexed). When speaker
diarization is
enabled, this identifies which speaker produced this utterance.
Speaker numbers
are consistent within a single connection but may vary between
different connections.
example: 1
language:
type: string
description: >
The automatically detected language code for this utterance (e.g.,
"en" for English,
"fr" for French).
example: en
RedactedAudioInfo:
type: object
required:
- start_ms
- duration_ms
properties:
start_ms:
type: integer
minimum: 0
description: >
Start time of the audio clip in milliseconds, relative to the
beginning
of the audio stream. This is the point from which the binary MP3
data begins.
example: 0
duration_ms:
type: integer
minimum: 0
description: Duration of the audio clip in milliseconds.
example: 3000