A residual block computes h_{t+1} = h_t + f(h_t). That's an Euler-method update rule for a differential equation: it's exactly the same form as taking one discrete step of size 1 along dh/dt = f(h, t). Neural ODEs (Chen et al., 2018) take this observation seriously: instead of stacking a fixed number of discrete residual layers, define the hidden state's evolution as a continuous-time differential equation dh/dt = f(h, t, θ), and let a numerical ODE solver decide how many steps to take and how big they should be.

The network f is a small neural network — it doesn't represent one layer, it represents the instantaneous rate of change of the hidden state at any point in time.

From discrete depth to continuous depth

In a normal network, "depth" means the number of stacked layers — a fixed, discrete integer you choose before training. In a Neural ODE, there's no fixed number of layers at all. You specify a starting time t0 and an ending time t1, and the output is whatever the ODE solver computes by integrating f from t0 to t1, starting at the input h(t0).

The solver (e.g., Runge-Kutta, Dopri5) adaptively decides how finely to step through time based on the accuracy it needs — in effect, the network dynamically chooses its own "effective depth" depending on how hard the particular input is to transform.

python

Note

Backpropagating through a deep stack of layers requires storing every intermediate activation — memory cost grows with depth. Neural ODEs instead use the adjoint sensitivity method: gradients are computed by solving a second, backward-in-time ODE, which needs only the final state and a constant amount of memory, regardless of how many solver steps were taken forward. This is the paper's headline result — effectively infinite depth at constant memory cost.

Why bother: irregular time series and exact invertibility

Two properties make Neural ODEs more than a theoretical curiosity:

Irregularly-sampled time series. Standard RNNs assume fixed, evenly-spaced time steps. Medical records, sensor logs, and financial ticks arrive at irregular intervals. A Neural ODE can integrate to any arbitrary timestamp t1 — "what was the hidden state at 3:47pm" — because the hidden state is a genuine continuous function of time, not a sequence of discrete, evenly-spaced updates.

Exact invertibility. Because an ODE's trajectory can be run forward or backward in time, continuous normalizing flows (a Neural-ODE-based generative model) get exact invertibility and exact likelihood computation almost for free — no need for the restrictive, carefully engineered layer structure that discrete normalizing flows require (see the next lesson).

Discrete layers vs. Neural ODEs

Property	Standard deep network	Neural ODE
Depth	Fixed integer, chosen upfront	Continuous — decided adaptively by the solver
Memory for backprop	Grows with number of layers	Constant, via the adjoint method
Handles irregular time steps	No — assumes fixed step size	Yes — natively
Inference cost	Fixed, predictable	Variable — depends on solver tolerance

Note

Neural ODEs are notably slower to train and run than equivalent discrete networks, since every forward pass requires calling an adaptive numerical solver, often taking many function evaluations. They are also finicky to train (stiff dynamics can make solvers take tiny, expensive steps) and have seen limited large-scale production adoption outside of specialized time-series and generative-modeling research. They remain mostly a research and specialized-application tool rather than a mainstream architecture choice.

Where they're actually used

Real use cases cluster around irregular time series in healthcare (patient vitals recorded at inconsistent intervals), physical system modeling where the underlying dynamics genuinely are continuous (fluid simulation, climate modeling), and continuous normalizing flows for density estimation and generative modeling. They have not displaced standard architectures in vision, language, or most tabular prediction tasks, where discrete layers are simpler, faster, and work just as well.

What's next

Neural ODEs showed that invertibility and continuous transformation can be first-class design goals. Normalizing Flows build on exactly this idea using purely discrete, carefully constructed invertible layers — trading the ODE solver's flexibility for speed and architectural simplicity.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →