Spiking Neural Networks (SNNs)

Real neurons don't output continuous numbers

Every network covered so far — MLPs, CNNs, transformers — uses continuous-valued activations: a neuron outputs a real number (0.73, -1.2, whatever) every single forward pass. Biological neurons don't work this way. A real neuron accumulates electrical charge and, when it crosses a threshold, fires a brief, discrete electrical pulse — a spike — then resets. Information in the brain is encoded largely in the timing and frequency of these spikes, not in a continuously varying number.

Spiking Neural Networks (SNNs) take this literally: instead of a neuron with a single continuous output, each neuron maintains an internal state over time and emits discrete spikes (binary events) when that state crosses a threshold.

The leaky integrate-and-fire neuron

The standard SNN neuron model is the Leaky Integrate-and-Fire (LIF) neuron. It maintains a membrane potential that accumulates incoming spikes (weighted by synaptic strength) over time, while continuously leaking — decaying back toward a resting value, like a capacitor discharging. When the membrane potential crosses a threshold, the neuron fires a spike and its potential resets.

This gives SNNs an inherent sense of time that standard networks lack: the same input pattern produces different output depending on when spikes arrive relative to each other, not just which neurons fired.

Why bother: energy efficiency and event-driven computing

The motivation for SNNs isn't biological realism for its own sake — it's energy efficiency. In a standard neural network, every neuron computes something on every forward pass, regardless of whether that computation matters. In an SNN, a neuron that isn't spiking does essentially nothing — computation only happens where and when spikes occur. On specialized neuromorphic hardware built to exploit this (Intel's Loihi chip, IBM's TrueNorth), this sparsity translates into dramatically lower power consumption than running an equivalent standard network on a GPU, because the hardware itself is event-driven rather than running dense matrix multiplications on every cycle regardless of activity.

Standard neural networks vs. Spiking Neural Networks

An honest assessment

SNNs occupy an unusual position: biologically well-motivated, genuinely useful for a narrow but real set of ultra-low-power applications, and still facing the same fundamental training difficulty (non-differentiable spikes) that has limited them for decades. Progress in surrogate-gradient training and neuromorphic hardware has narrowed the accuracy gap for some tasks, but SNNs are best understood today as a specialized tool for energy-constrained, event-driven settings rather than a general replacement for standard deep learning.

What's next

This closes out a tour of architectures that depart from the mainstream CNN/transformer path in different ways: structural (GNNs, Capsules), mathematical (Normalizing Flows, Neural ODEs, EBMs), or biological (RBMs, SOMs, SNNs). Revisit the Transformers and Autoencoders lessons to see how the dominant architectures handle the same underlying problems — representation, generation, and structure — that these alternative approaches tackle from a different angle.