Audio Generation Models
Why raw audio is a hard generative target
A single second of CD-quality audio is 44,100 raw samples. Generating a plausible few seconds of audio one raw sample at a time, autoregressively, means modeling a sequence hundreds of thousands of steps long — orders of magnitude longer than a typical text sequence. Every major audio generation architecture is, in large part, a strategy for compressing this problem down to a tractable sequence length before applying standard generative modeling techniques (autoregression, diffusion) on top.
Neural audio codecs: compressing audio into discrete tokens
Models like EnCodec and SoundStream solve the sequence-length problem by learning a codec: an encoder compresses raw audio into a much shorter sequence of discrete tokens (via residual vector quantization — multiple codebooks, each refining the quantization error left by the previous one), and a decoder reconstructs audio from those tokens. Once a good codec exists, audio generation reduces to the already-solved problem of generating a sequence of discrete tokens, using the exact same transformer architectures used for text — this is the approach behind MusicGen and AudioLM.
import numpy as np
def residual_vq_encode(vector, codebooks):
"""codebooks: list of (K, D) codebook arrays, one per quantization stage.
Each stage quantizes the residual left by the previous stage."""
residual = vector.copy()
indices = []
for codebook in codebooks:
distances = np.linalg.norm(codebook - residual, axis=1)
idx = int(np.argmin(distances))
indices.append(idx)
residual = residual - codebook[idx] # what THIS stage failed to capture
return indices # a short list of discrete token ids representing the vector
# More codebooks/stages = finer reconstruction, at the cost of more tokens per audio frame.Discrete tokens turn audio generation into a language-modeling problem
Diffusion-based alternatives
The other major family (used by models like Stable Audio and some versions of MusicLM) applies diffusion directly to a compressed spectrogram or latent audio representation, following the same denoising recipe as image diffusion: start from noise, iteratively denoise conditioned on a text prompt, decode the final latent back to a waveform. This trades the sequential, token-by-token generation cost of autoregressive codec models for the iterative-but-parallelizable-per-step cost of diffusion, and tends to produce longer coherent segments more easily since diffusion isn't locked into a strict left-to-right generation order.
Autoregressive codec-token models vs diffusion audio models
| Property | Autoregressive (MusicGen, AudioLM) | Diffusion (Stable Audio) |
|---|---|---|
| Generation order | Strictly sequential, token by token | Iterative denoising, whole-clip at once per step |
| Long-range coherence | Can drift/degrade over long sequences | Generally more stable over longer clips |
| Inference latency | Scales with output length | Scales with number of denoising steps, not length |
| Underlying representation | Discrete codec tokens | Continuous latent spectrogram-like representation |
Evaluation is still largely subjective
What's Next?
The codec-token-as-language-model pattern established here is likely to keep converging further with general-purpose multimodal LLMs, which increasingly treat audio tokens as just another modality alongside text and image tokens inside a single unified generative model.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.