Realtime events

Realtime STT exchanges JSON events after the WebSocket connection is open and configured. This page describes the transcript events and finalization flow.

Partial transcript

{
  "type": "conversation.item.input_audio_transcription.delta",
  "event_id": "evt_...",
  "item_id": "item_...",
  "content_index": 0,
  "delta": "Hello",
  "text": "[low]Hello[/low]",
  "enriched_text": "[low]Hello[/low]",
  "clean_text": "Hello",
  "accent": "",
  "tags": [
    {"type": "tag", "label": "low", "start": 0, "end": 5},
    {"type": "tag", "label": "low", "start": 10, "end": 16}
  ],
  "is_final": false,
  "session_id": "<request-id>",
  "request_id": "<request-id>"
}

Field	Meaning
`delta`	OpenAI-compatible clean suffix when the hypothesis only extends
`clean_text`	Current cumulative clean transcript snapshot
`text`	Current cumulative enriched transcript
`enriched_text`	Explicit alias for `text`
`accent`	Best-effort accent classification; may be empty or change
`tags`	Parsed ranges into the current `enriched_text`
`item_id`	Correlates partial and completed events for one turn
`request_id`	Session trace identifier

Partial transcripts are mutable drafts. For a live UI, replace the displayed clean and enriched partial with each cumulative clean_text and enriched_text snapshot. These snapshots are cumulative within the current turn, not across every turn in a multi-turn WebSocket session.

delta is provided for integrations that append only new clean text. It may be empty when the model revises an earlier word. In that case, clean_text contains the useful replacement snapshot.

Completed transcript

{
  "type": "conversation.item.input_audio_transcription.completed",
  "event_id": "evt_...",
  "item_id": "item_...",
  "content_index": 0,
  "transcript": "Hello there.",
  "text": "[low]Hello there.[/low]",
  "enriched_text": "[low]Hello there.[/low]",
  "clean_text": "Hello there.",
  "accent": "General American",
  "segments": [
    {
      "segment_id": 1,
      "segment_start_ms": 100,
      "segment_end_ms": 1300,
      "pre_roll_ms": 100,
      "trailing_silence_ms": 0,
      "pending_pause_ms_before_segment": 100,
      "emitted_pause_tag": "",
      "text": "[low]Hello there.[/low]"
    }
  ],
  "tags": [
    {"type": "tag", "label": "low", "start": 0, "end": 5},
    {"type": "tag", "label": "low", "start": 17, "end": 23}
  ],
  "is_final": true,
  "session_id": "<request-id>",
  "request_id": "<request-id>"
}

The completed event is authoritative. Replace any provisional UI state with:

transcript or clean_text for final plain text.
enriched_text or text for final emotion-aware text.
accent, tags, and segments from the completed event.

Realtime segments use millisecond fields such as segment_start_ms and segment_end_ms. This differs from upload verbose_json, where segment timestamps are start and end values in seconds.

Segment timing fields

pre_roll_ms and pending_pause_ms_before_segment both describe time before a detected speech segment, but they measure different things:

Field	Meaning
`pre_roll_ms`	Audio retained from immediately before speech was detected and included as model context. It helps prevent the first sound or syllable from being clipped.
`pending_pause_ms_before_segment`	The complete silence or non-speech gap measured before the segment. The server can use it to produce an `emitted_pause_tag` such as `[pause_short]` or `[pause_medium]`.

For example, after a two-second gap the server may retain only the final 350 ms as model context:

{
  "pre_roll_ms": 350,
  "pending_pause_ms_before_segment": 2000,
  "emitted_pause_tag": "[pause_medium]"
}

The prefix_padding_ms session setting controls the maximum desired pre-roll; pre_roll_ms reports how much audio was actually available and retained for a particular segment. These segment fields are returned diagnostics. Clients do not send them in audio append events.

Enriched markup

Enrichment is model output and can change while a transcript is partial.

Form	Meaning
`[label]speech[/label]`	Emotion, tone, or delivery label over a span
`[pause_short]`, `[pause_medium]`, `[pause_long]`	Audible pause
`(word)`, `((word))`, `(((word)))`	Increasing emphasis or stress

Treat labels as model output rather than a closed enum.

Each tags entry uses a half-open range into that event's enriched_text: start is inclusive and end is exclusive. Ranges may move between partials, so replace old tags whenever a new snapshot arrives.

Manual turn flow

With turn_detection: null:

session.created
→ session.update
→ session.updated
→ input_audio_buffer.append ...
→ transcript delta ...
→ input_audio_buffer.commit
→ input_audio_buffer.committed
→ transcript completed

Server VAD flow

With turn_detection.type: "server_vad", the API can also emit:

input_audio_buffer.speech_started
input_audio_buffer.speech_stopped
input_audio_buffer.committed

An explicit commit remains useful to flush active speech when a stream ends.

Error event

Errors after the WebSocket upgrade use this shape:

{
  "type": "error",
  "event_id": "evt_...",
  "error": {
    "type": "authentication_error",
    "code": "invalid_key",
    "message": "Invalid API key.",
    "param": null
  },
  "request_id": "<request-id>"
}

KittenML completes the WebSocket upgrade before sending authentication and admission errors. The current public endpoint then exposes close code 1000. Use the structured error.code—not the close code—to distinguish an invalid key, exhausted capacity, or a temporary service failure.

Use error.code for program logic, retain request_id, and reconnect as a new session only when the failure is transient.

Partial transcript​

Completed transcript​

Segment timing fields​

Enriched markup​

Manual turn flow​

Server VAD flow​

Error event​