Voice AI Engineer Interview: STT Latency, Interrupt...

A voice assistant answers quickly but talks over the user, truncates pauses, repeats a tool action after reconnecting, and occasionally responds to a partial transcript. Calling it “low latency” does not make the experience reliable.

Voice AI interviews test the entire turn loop: capture, transport, endpointing, transcription, reasoning, tools, synthesis, playback, interruption, and recovery. The strongest answer treats latency as one budget within a stateful conversation, not as one vendor number.

Draw the End-to-End Voice Turn Loop

Start from the user's acoustic input and end at audible, correct response. Name every stage that can delay or distort the turn:

microphone capture and device processing;
media encoding and network transport;
server ingress and buffering;
voice activity detection or endpointing;
partial and final transcription;
context assembly, model reasoning, and tools;
text-to-speech generation;
return transport, buffering, and playback.

Mark which stages stream and which wait for completion. A system can begin transcription while audio arrives and begin synthesis before the entire answer is generated. Streaming reduces perceived delay but creates cancellation and reconciliation complexity.

The real-time STT and LLM answer-quality guide explains why input quality and turn boundaries affect downstream answers. A voice engineer must also own transport, playback, and state behavior.

Do not assume the measured server response equals user-perceived latency. Instrument the device-to-device path and include the time until the first useful audible content, not only the first internal token.

Allocate a Latency Budget Without Inventing a Universal Threshold

There is no universal acceptable voice latency. A rapid interruption-sensitive assistant, a language tutor, a medical intake flow, and a voice-controlled transaction have different tolerance, risk, and confirmation needs.

Define the product scenario and measure distributions rather than one average. Useful segments include time to speech detection, endpoint decision, first transcript, final transcript, model start, first synthesized audio, and complete response. Track median and tail behavior across network, device, language, noise, and turn length.

Then decide where waiting buys quality. A longer endpoint window may preserve a thoughtful pause but delay response. Early model execution may improve speed but use an unstable partial transcript. Larger audio buffers may smooth playback but make interruption feel slow.

Set a budget from user experience and consequence. State the test conditions whenever you quote a number. Do not compare vendor latency or accuracy unless the audio, network, language, hardware, configuration, and measurement method are matched.

OpenAI Realtime, Twilio Media Streams, Deepgram endpointing, and WebRTC documentation describe different parts of the stack. Their presence does not establish one end-to-end performance result.

Choose Endpointing and Turn Detection Deliberately

Silence is not always the end of a turn. People pause to think, breathe, search for a word, or wait for a screen. Background noise can also make continuous speech appear active.

Endpointing policy balances responsiveness against truncation. Consider:

silence duration and recent speech confidence;
punctuation or semantic completeness from partial text;
language and speaking style;
user-controlled push-to-talk or explicit commit;
timeout behavior for abandoned turns;
repair when the endpoint was wrong.

Do not hide the policy. A subtle listening or processing state helps users understand whether the system expects more speech. For high-consequence actions, explicit confirmation may be safer than relying on inferred turn completion.

Deepgram's endpointing documentation is one vendor's implementation guidance, not a universal rule. Explain the principle and how you would validate it with representative speech, interruptions, pauses, accents, noise, and task context.

The live transcript noise-control guide provides a practical taxonomy for partial questions, speaker confusion, technical-term errors, and stale fragments.

Handle Barge-In as Coordinated Cancellation

Barge-in occurs when the user speaks while the assistant is producing audio. Stopping the speaker output is only the visible first step.

The system must decide what happens to:

buffered audio not yet played;
text-to-speech generation;
model generation;
tool calls already proposed or executing;
assistant text already committed to history;
user audio that overlaps with playback;
downstream actions that cannot be canceled.

Use explicit cancellation identifiers and state transitions. Mark how much assistant audio the user actually heard, not merely how much was generated. Feed the next turn only the committed history.

For a read-only response, canceling generation may be enough. For a tool that sends a message or changes an account, interruption cannot undo an executed side effect. The system needs idempotency, status checks, and a user-visible confirmation of what occurred.

Test rapid repeated interruptions and race conditions. A stale completion should not resume playback after a newer turn begins.

Reconcile Partial Transcripts and Conversation State

Streaming STT can revise earlier words. The partial transcript that starts model work may differ from the final transcript. Treat partial content as provisional state.

Maintain identifiers for the user turn, transcript revision, model response, tool action, and assistant playback. When the transcript changes materially, decide whether to cancel, restart, or continue with a correction. The choice depends on how far the response progressed and whether external action began.

Separate acknowledged state from speculative state. A user should not hear an answer to a version of the question the system later discarded without some repair.

When confidence is low or two interpretations produce different actions, ask a narrow clarification. Do not silently fill missing words from context and present the result as certain.

The missed-question recovery loop is candidate-facing, but its central lesson applies: reconstruct the real prompt before optimizing the answer. In a production system, the reconstruction must be represented in state and receipts.

Recover From Transport, Model, and Tool Failures

Classify the failure before choosing recovery. Transport loss, STT degradation, model timeout, synthesis error, and tool failure require different responses.

A safe recovery flow should:

identify the last acknowledged user and assistant state;
stop obsolete generation and playback;
check whether a tool action executed;
reconnect or switch to an approved fallback;
tell the user what was retained or lost;
ask for a narrow replay or confirmation when needed;
prevent duplicate external actions.

Avoid replaying a full assistant response automatically after reconnection. The user may have already heard part of it or changed intent. Restore from explicit state, not from a guess based on buffered content.

The agent observability interview guide explains how traces and failure receipts can connect voice turns, model calls, tools, and state transitions without logging sensitive content by default.

Measure Quality Beyond Word Error Rate

Word error rate can be useful under a defined matched test, but it does not capture every conversational failure. A transcript can contain few word errors and still miss a negation, speaker, entity, or turn boundary that changes the action.

Build a measurement set around the product:

end-to-end and segment latency distributions;
endpoint truncation and delayed-response rates;
interruption success and stale-playback rate;
clarification rate;
duplicated or inconsistent tool action rate;
task completion and user correction burden;
failure recovery success by network and device slice.

Preserve representative audio only when allowed and necessary. Apply consent, minimization, redaction, access controls, and retention limits.

Pair automated signals with human review of difficult slices. Do not publish unmatched vendor accuracy comparisons or treat one clean-room audio set as proof of field performance.

Answer a Voice AI System Scenario

Use seven moves: user scenario, turn loop, budget, endpoint policy, interruption state, failure recovery, and measurement.

A concise answer might say: “I would first map capture through playback and measure each segment on representative devices and networks. I would set endpointing from the conversation style rather than a universal silence value. Partial transcripts stay provisional. Barge-in cancels playback and obsolete generation through turn identifiers, while tool execution is checked separately for completed side effects. On reconnect, the system restores the last acknowledged state, prevents duplicate actions, and tells the user what needs repetition. I would evaluate tail latency, truncation, interruption races, task completion, and recovery by slice.”

State the limits: WebRTC does not guarantee low latency, one vendor's endpoint defaults do not fit every product, and stopping audio does not undo an external action.