Self-hosted voice AI: Whisper + neural TTS in production, without GPUs

The short answer: you do not need a GPU, and you do not need a per-minute transcription API, to put voice in production in 2026. Quantized Whisper models transcribe on ordinary CPUs fast enough for messaging-style workloads, and neural text-to-speech is effectively free. The engineering that matters is packaging, latency budgeting, and the async plumbing around it — not the models.

Speech-to-text: Whisper on CPU is enough

The faster-whisper runtime with int8 quantization changes the economics: the small multilingual model transcribes a typical 5–10 second voice note in 5–15 seconds on two ordinary vCPUs, in ~1–2 GB of RAM — and it handles code-switched speech (Urdu/English in one sentence) credibly. Sizing rule of thumb: small for messaging bots, medium when accuracy complaints start, large only with a GPU. One production note: multilingual models sometimes label spoken Urdu as Hindi — treat them as the same voice pathway and your users won’t notice.

Text-to-speech: neural voices are a solved problem

Free neural TTS engines now produce voices that customers happily listen to, in dozens of languages. Two lessons from production: feed the engine native script (real Urdu, not romanized transliteration) — pronunciation improves dramatically — and slow the speaking rate slightly; a calm pace reads as a person talking with someone rather than an announcement read at them. Convert output to OGG/Opus and messaging platforms render it as a normal voice note.

The architecture that survives production

• Bake the model into the container image. A runtime download means your first user waits minutes. A pre-fetched model is a bigger image (~500 MB) and a instant first response — the right trade every time.
• Never transcribe inside a webhook request. Platforms expect an acknowledgement in seconds; transcription takes tens of seconds. Ack immediately, process in a background task, deduplicate by message id — or your users get double replies.
• Echo the transcript back (“I heard: …”) before acting on it. Mishearings happen; in anything consequential, the user must get the chance to correct the machine before the machine decides.
• Keep a text copy authoritative. Voice is the interface; the logged text is the record.

When managed services win

At high volume, spiky load, or hard latency floors, managed transcription (Amazon Transcribe, Azure Speech) and premium TTS (ElevenLabs — noticeably better in smaller languages) earn their per-minute pricing. Below roughly a few thousand voice messages a day, self-hosted on one modest server wins on cost, privacy and control — which is exactly where most businesses adding voice actually are.