Temporal Convolutional Networks

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

Convolutions as an alternative to recurrence

Recurrent models process a sequence one step at a time, which makes long sequences slow to train and historically prone to vanishing gradients. Temporal Convolutional Networks (TCNs), formalized by Bai, Kolter, and Koltun (2018) in 'An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling,' take a different approach: apply 1D convolutions across the time axis, the same way CNNs apply 2D convolutions across image pixels. A convolution can be computed for every time step in parallel, and the authors showed that, with the right architectural tricks, convolutional models matched or beat LSTMs and GRUs across a range of sequence tasks.

Causal convolutions: no peeking at the future

A standard convolution centers its filter on each position, which means it would use future time steps to predict the present — not allowed in forecasting. TCNs use causal convolutions: the output at time $t$ is computed only from inputs at time $t$ and earlier. This is implemented by padding the input on the left (past side) by (kernel_size − 1) × dilation and trimming the output so nothing from $t+1$ onward ever leaks into the computation for $t$.

Dilated convolutions: exponentially growing receptive field

A single causal convolution with kernel size $k$ only looks back $k-1$ steps — far too short to capture long-range patterns on its own. Stacking plain convolutions linearly increases the receptive field with depth, requiring many layers to see far back. TCNs instead use dilated convolutions: layer $i$ uses dilation factor $d_i = 2^i$, skipping over $d_i - 1$ inputs between each kernel tap. The receptive field after $L$ layers with kernel size $k$ grows as $1 + 2(k-1)(2^L - 1)$ — exponential in depth, so a modest number of layers can cover a receptive field of hundreds or thousands of time steps.

Formally, a dilated causal convolution at dilation $d$ is $F(t) = \sum_{i=0}^{k-1} f(i) \cdot x_{t - d \cdot i}$, where $f$ is the learned filter.

A minimal TCN residual block in PyTorch

python

import torch
import torch.nn as nn

class CausalConv1d(nn.Module):
    def __init__(self, in_ch, out_ch, kernel_size, dilation):
        super().__init__()
        self.pad = (kernel_size - 1) * dilation
        self.conv = nn.Conv1d(in_ch, out_ch, kernel_size,
                               padding=self.pad, dilation=dilation)

    def forward(self, x):
        out = self.conv(x)
        return out[:, :, :-self.pad] if self.pad != 0 else out  # trim future leakage

class TCNBlock(nn.Module):
    def __init__(self, in_ch, out_ch, kernel_size, dilation, dropout=0.2):
        super().__init__()
        self.conv1 = CausalConv1d(in_ch, out_ch, kernel_size, dilation)
        self.conv2 = CausalConv1d(out_ch, out_ch, kernel_size, dilation)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(dropout)
        self.downsample = nn.Conv1d(in_ch, out_ch, 1) if in_ch != out_ch else None

    def forward(self, x):
        out = self.dropout(self.relu(self.conv1(x)))
        out = self.dropout(self.relu(self.conv2(out)))
        residual = x if self.downsample is None else self.downsample(x)
        return self.relu(out + residual)   # skip connection, like a ResNet block

class TCN(nn.Module):
    def __init__(self, in_ch, channels=(32, 32, 32, 32), kernel_size=3):
        super().__init__()
        layers = []
        prev_ch = in_ch
        for i, ch in enumerate(channels):
            layers.append(TCNBlock(prev_ch, ch, kernel_size, dilation=2 ** i))
            prev_ch = ch
        self.network = nn.Sequential(*layers)
        self.head = nn.Conv1d(prev_ch, 1, 1)

    def forward(self, x):          # x: (batch, channels, time)
        return self.head(self.network(x))

Residual connections make deep TCNs trainable

Each TCN block wraps its convolutions in a residual (skip) connection, exactly like a ResNet block: the block learns a correction to add to its input rather than having to reproduce the input from scratch. This lets gradients flow directly through the skip path during backpropagation, which is what makes it practical to stack enough dilated layers to get a large receptive field without the network becoming hard to train.

TCN vs. LSTM for sequence modeling

Property	TCN	LSTM
Training parallelism	Full — all time steps computed at once	Sequential — one step at a time
Receptive field	Fixed, set by depth/dilation/kernel size	Theoretically unbounded (via cell state)
Memory usage during training	Scales with sequence length x channels	Lower, but sequential compute dominates
Gradient stability	Stable (convolutions + residuals)	Improved by gating, but still can vanish over very long spans

The receptive field is fixed at design time

Unlike an LSTM, whose cell state could in principle carry information from the very first time step, a TCN's receptive field is a fixed, computable number determined by kernel size, dilation schedule, and depth. If a dependency in your data lies outside that window, the model architecturally cannot see it — you have to deliberately size kernel/dilation/depth to cover the longest dependency you expect, which requires some domain knowledge or experimentation up front.

What's next

TCNs demonstrate that convolution plus dilation plus residual connections can rival recurrence for sequence modeling, with much better training throughput. Both TCNs and LSTMs, however, model a series' internal dynamics directly — neither is designed to flag when the dynamics themselves abruptly change. Detecting those regime shifts is a distinct problem with its own dedicated methods.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →