Speech: STT and TTS
The audio layer under voice AI products. One of the roles in scope names “AI solutions for communication, audio/video streaming, chatbots”, which is this plus 02_realtime_voice_agents.md.
Speech to text
Modern ASR is a sequence model over audio features. What matters in an interview is the operational shape, not the architecture.
Batch versus streaming. Batch transcription sends a complete file and returns the full transcript — higher accuracy, because the model sees the whole context, and unsuitable for conversation. Streaming emits partial hypotheses as audio arrives, revising them as more context appears. The revision behaviour is the part that surprises people: an interim transcript can change before it is final, so downstream logic must key on final results.
Metrics. Word Error Rate (substitutions + insertions + deletions, over reference words) is the standard, and it is a blunt instrument. WER weights every word equally, so getting “not” wrong scores the same as getting “the” wrong. For a product, measure what matters: entity accuracy on names, numbers and drug names; intent accuracy after the transcript reaches the LLM.
Where accuracy actually degrades: accents and dialects, domain vocabulary, overlapping speakers, telephone-band audio (8 kHz narrowband loses a lot), and background noise. Vendor WER figures are quoted on clean read speech and do not survive a call centre.
Features you will need to name:
| Feature | For |
|---|---|
| Diarization | who spoke when — required for multi-party transcripts |
| Word-level timestamps | aligning transcript to audio, highlighting, editing |
| Custom vocabulary / biasing | product names, drug names, SKUs the base model has never seen |
| Punctuation and formatting | raw ASR output is an unpunctuated stream |
| VAD (voice activity detection) | segmenting speech from silence; the input to turn detection |
| Language identification | multilingual deployments |
Custom vocabulary is the highest-leverage fix in most real deployments. A domain term transcribed wrong every time destroys downstream extraction, and biasing costs far less than fine-tuning.
Options: hosted APIs from the major cloud and speech vendors, or self-hosted open-weights models (the Whisper family and its faster reimplementations) when data cannot leave your boundary or volume makes per-minute pricing painful. Self-hosting is genuinely viable here — unlike frontier LLMs, good ASR runs on modest GPUs.
Text to speech
Quality axes: naturalness, latency, controllability (pacing, emphasis, pronunciation via SSML or phoneme hints), and voice cloning. Cloning carries consent and likeness obligations — the EU AI Act’s transparency rules require disclosure when a user is interacting with synthetic media, in force since 2 August 2026. See ../14_guardrails_safety/.
Streaming TTS is the requirement for conversation. Synthesising a whole reply before playback adds its full duration to perceived latency. Stream audio chunks as they are generated, and start synthesis on the first sentence rather than waiting for the LLM to finish.
Time to first byte is the metric that matters, not total synthesis time. A voice that starts in 150 ms and finishes slowly feels responsive; one that starts in 900 ms does not, regardless of throughput.
Cost and architecture
Audio is billed per minute or per character, not per token, which changes the optimisation:
- Cache TTS for anything repeated — prompts, menu options, confirmations. A surprising share of a voice product’s speech is fixed strings.
- Do not transcribe silence. VAD-gate the stream before it reaches the ASR.
- Choose the transport deliberately: WebRTC for live conversation, WebSocket for server-to-server streaming, plain HTTP for batch. See ../../backend/12_protocols/websockets/.
- Store audio only if you need it, and if you do, treat it as personal data with a retention policy. Voice is biometric-adjacent under GDPR and recordings in a regulated setting carry the same obligations as any other record. See ../../backend/31_healthcare_regulated/02_phi_privacy_and_secure_coding.md.
Testing a speech pipeline
- A held-out audio set with reference transcripts, including the hard cases — accents, noise, domain vocabulary — not just clean samples.
- Track WER and the downstream metric, because the two diverge. A model with worse WER that gets entity names right is the better product choice.
- Regression-test on every model or prompt change; providers update models under the same endpoint name, which silently changes your accuracy.
- Synthetic audio for load testing, real audio for accuracy.
Interview angle
- “Streaming or batch transcription?” - streaming for conversation, accepting lower accuracy and interim results that get revised; batch when the whole file is available and accuracy matters more than latency. The gotcha is that interim transcripts change, so downstream logic must act on finals.
- “Why is WER a poor product metric?” - it weights every word equally. Getting a negation or a drug name wrong is catastrophic; getting an article wrong is not. Measure entity-level accuracy and the downstream task metric alongside it.
- “Transcription quality is poor on our domain terms - what do you do first?” - custom vocabulary or keyword biasing. It is cheap, immediate, and fixes the specific failure. Fine-tuning is a much larger commitment for a problem biasing usually solves.
- “What latency metric matters for TTS?” - time to first audio byte, not total synthesis time. Stream chunks and start synthesising on the first sentence rather than waiting for the full LLM response.
- “Would you self-host speech models?” - plausibly yes, unlike frontier LLMs. Good open-weights ASR runs on modest hardware, and self-hosting answers both the data-residency question and the per-minute cost at volume.