Text-to-Speech (TTS) takes written text and produces an audio waveform of someone speaking it. It looks like the mirror image of speech recognition, and architecturally it often is, but it's arguably a harder generation problem: ASR only has to produce one correct answer (the words that were actually said), while TTS has to invent an entire acoustic signal — pitch, rhythm, emphasis, breathing, pacing — that doesn't exist anywhere in the input text. The same sentence can be spoken in infinitely many valid ways, and a TTS system has to pick one that sounds natural rather than robotic.

The classical two-stage pipeline

Older TTS systems, like older ASR systems, split the problem into stages. Concatenative synthesis stitched together pre-recorded snippets of real human speech, selected to match the target text — intelligible, but prone to audible seams and unnatural prosody. Parametric synthesis modeled speech statistically and generated it from scratch using a vocoder, producing smoother but noticeably synthetic, "robotic" audio — the classic text-to-speech sound most people associate with early GPS navigation and screen readers.

Neural TTS changed both stages. Modern systems typically still use a two-stage pipeline, but each stage is now a trained neural network rather than hand-tuned signal processing rules.

Stage 1: text to mel spectrogram

The first neural stage takes normalized text (expanded abbreviations, numbers spelled out, etc.) and predicts an intermediate acoustic representation — almost always a mel spectrogram, the same time-frequency representation used on the ASR side. This stage has to make all the prosodic decisions: how long each phoneme should last, what pitch contour the sentence should follow, where emphasis falls. Models like Tacotron and FastSpeech use attention or duration-prediction mechanisms to align input text tokens to output spectrogram frames, since a word can stretch over many frames while being represented by just a few text tokens.

Stage 2: mel spectrogram to waveform (the vocoder)

A spectrogram only encodes magnitude information per frequency — it discards the precise waveform phase needed to actually produce sound through a speaker. The vocoder is a model trained specifically to reconstruct a realistic, high-fidelity waveform from a spectrogram. Early neural vocoders like WaveNet generated audio sample-by-sample autoregressively, which was high quality but extremely slow. Later vocoders (WaveGlow, HiFi-GAN) generate audio in parallel, trading a small amount of quality for orders-of-magnitude faster generation — critical for any real-time voice assistant.

text

Note

Just as Whisper collapsed ASR's separate acoustic and language models into one transformer, newer TTS systems (like VITS and various diffusion- and transformer-based approaches) train the text-to-spectrogram and spectrogram-to-waveform stages jointly, or skip the explicit spectrogram step entirely and generate waveforms (or discrete audio tokens) directly from text. This removes the error-compounding problem of stage one's mistakes propagating into stage two, and tends to produce more naturally expressive speech — at the cost of needing much larger training datasets, the same tradeoff seen across neural audio modeling generally.

Why modern neural TTS sounds human

Factor	Older parametric TTS	Modern neural TTS
Prosody (rhythm, pitch, emphasis)	Hand-tuned rules, limited variety	Learned from thousands of hours of expressive human speech
Vocoder quality	Simple signal-processing vocoders, audibly synthetic	Neural vocoders trained to match real waveform statistics
Voice variety	One fixed voice per system, costly to add new ones	Voice cloning from minutes (or seconds) of reference audio
Naturalness under long sentences	Degrades, flat intonation	Maintains contextual intonation across full sentences

Voice cloning and conditioning

Many modern TTS systems can be conditioned on a short reference audio clip of a target speaker, producing synthetic speech in that person's voice without retraining the whole model — done by extracting a compact speaker embedding from the reference audio and feeding it alongside the text input. This capability is powerful for accessibility and personalization, and is also why voice cloning raises real consent and misuse concerns: the same mechanism that lets a user restore their own voice after illness can, without safeguards, impersonate someone else's.

What's next

ASR and TTS cover the audio-text boundary for speech specifically. Computer vision tackles a parallel boundary for images — continue to Computer Vision: Object Detection to see how models locate and identify objects within an image, building on the convolutional and transformer ideas already covered.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →