Speech-to-Text Model Architectures

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

The core challenge: aligning audio to text

Automatic speech recognition (ASR) converts an audio waveform into text. The fundamental difficulty is alignment: an audio signal sampled at 16kHz produces thousands of raw samples per second, but the corresponding transcript might be only a few words long, and there's no explicit marker for which audio samples correspond to which letters or phonemes. Every modern ASR architecture is, at its core, a different strategy for learning this audio-to-text alignment without ever being given it explicitly during training.

From raw audio to a spectrogram

Almost no ASR model operates on raw waveform samples directly. The standard preprocessing step converts audio into a mel spectrogram: a 2D representation with time on one axis and frequency (on a perceptually-motivated mel scale, matching human pitch sensitivity) on the other, with intensity indicating how much energy is present at each time-frequency bin. This representation compresses the raw signal (often 16,000 samples/second) down to a much shorter sequence of frames (commonly 100/second) while preserving the acoustic information that actually matters for recognizing speech, making it tractable as a sequence model input.

CTC: alignment-free sequence labeling

Connectionist Temporal Classification (CTC), used in architectures like DeepSpeech and wav2vec 2.0's fine-tuning head, sidesteps explicit alignment entirely. The model outputs one probability distribution over characters/tokens (plus a special blank symbol) for every input time frame, producing a much longer sequence than the target transcript. The CTC loss sums the probability over every possible way of collapsing that long sequence (by merging repeated characters and removing blanks) into the correct target transcript, computed efficiently via a dynamic-programming forward-backward algorithm — conceptually similar to the one used in HMMs. This lets the model learn alignment implicitly from paired (audio, transcript) data alone.

ctc_collapse.py

python

def ctc_collapse(frame_predictions, blank_token="_"):
    """Collapse a per-frame CTC output sequence into the final transcript:
    merge consecutive repeats, then drop blank tokens."""
    collapsed = []
    prev = None
    for token in frame_predictions:
        if token != prev:
            collapsed.append(token)
        prev = token
    return [t for t in collapsed if t != blank_token]

# Example: model predicts "h h _ e e _ l l l _ _ o o" frame by frame
frames = ["h", "h", "_", "e", "e", "_", "l", "l", "l", "_", "_", "o", "o"]
print("".join(ctc_collapse(frames)))  # -> "hello"

Encoder-decoder (attention) models learn alignment implicitly too, differently

Whisper and other sequence-to-sequence ASR models use a transformer encoder over the audio spectrogram and an autoregressive transformer decoder that generates text tokens one at a time, attending back over the full encoded audio at every decoding step via cross-attention. Unlike CTC, there's no explicit frame-by-frame output or blank-token collapsing — the decoder's attention weights implicitly learn which audio region corresponds to which output token, and the model can be trained with plain next-token cross-entropy, just like a language model, making it straightforward to also handle translation and other side tasks within the same architecture.

CTC vs encoder-decoder (attention) ASR

Property	CTC-based (wav2vec 2.0, DeepSpeech)	Encoder-decoder (Whisper)
Output generation	Parallel, one distribution per frame	Autoregressive, one token at a time
Alignment assumption	Monotonic (strictly left-to-right)	Learned via cross-attention, flexible
Inference speed	Fast — single forward pass	Slower — sequential decoding
Extra capabilities	Transcription only	Can also translate, detect language, timestamp

Word error rate hides where errors actually occur

ASR systems are benchmarked on Word Error Rate (WER) — substitutions + insertions + deletions, divided by the number of reference words. A model can post an impressively low aggregate WER while still failing badly on rare proper nouns, accented speech, overlapping speakers, or domain-specific jargon, since these errors are diluted across a large, mostly-easy evaluation set. Production deployments typically track WER broken down by subgroup (accent, domain, noise condition) rather than trusting one global number.

What's Next?

The same spectrogram-plus-transformer recipe that powers transcription also underlies the reverse problem — synthesizing speech from text — where the challenge flips from compressing audio into text to expanding text into a convincing, natural-sounding acoustic signal.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →