Streaming Data ML Pipelines

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

From batch jobs to always-on pipelines

A traditional ML pipeline runs on a schedule: extract a batch of data, transform it, train or score, write results, repeat tomorrow. A streaming ML pipeline instead processes events continuously as they arrive — a purchase, a sensor reading, a page view — computing features and predictions within milliseconds to seconds of the event occurring. This isn't just 'batch but more often'; it requires fundamentally different infrastructure for state management, windowing, and exactly-once processing guarantees.

Core building blocks

Streaming pipelines are typically built on a log-based message broker (Kafka, Pulsar, Kinesis) that durably stores an ordered, replayable stream of events, paired with a stream processing engine (Flink, Spark Structured Streaming, Kafka Streams) that consumes the log and computes transformations continuously. The broker decouples producers from consumers and provides replay — critical for recovering from processing failures without losing or duplicating data. The processing engine maintains stateful operators (running aggregates, joins, windowed computations) that persist across events, checkpointed periodically so state survives restarts.

Windowing: the key abstraction for streaming features

Most streaming features are aggregates over a time window: 'transactions per user in the last 10 minutes,' 'rolling average latency over 5 minutes.' Three window types dominate: tumbling windows (fixed, non-overlapping intervals, e.g., every 5 minutes), sliding windows (overlapping, recomputed on every new event, giving smoother but more compute-intensive aggregates), and session windows (dynamically sized, closing after a gap of inactivity — natural for user-session-based features). A critical subtlety is event time vs. processing time: event time is when something actually happened; processing time is when the pipeline saw it. Network delays and out-of-order arrival mean these diverge, and engines like Flink use watermarks — a heuristic estimate of 'we've seen all events up to time T' — to decide when a window can be safely closed and emitted.

Window type	Boundary behavior	Typical use
Tumbling	Fixed, non-overlapping intervals	Per-minute aggregate metrics
Sliding	Overlapping, recomputed per new event	Smooth rolling averages for real-time features
Session	Closes after a gap of inactivity	User session length, engagement bursts

streaming_feature_sketch.py

python

# Pseudocode illustrating a sliding-window feature computation
# using a stream processing API similar to Flink's Python (PyFlink) DataStream API.

from pyflink.datastream import StreamExecutionEnvironment
from pyflink.datastream.window import SlidingEventTimeWindows
from pyflink.common import Time

env = StreamExecutionEnvironment.get_execution_environment()

events = env.from_source(kafka_source, watermark_strategy, "transactions")

rolling_spend = (
    events
    .key_by(lambda e: e.user_id)
    .window(SlidingEventTimeWindows.of(Time.minutes(10), Time.minutes(1)))
    .reduce(lambda a, b: a.with_amount(a.amount + b.amount))
)

# This produces, once per minute, each user's trailing-10-minute spend —
# usable directly as a real-time fraud-detection feature.
rolling_spend.sink_to(feature_store_sink)

Training-serving skew is the recurring failure mode

The hardest problem in streaming ML isn't the stream processing itself — it's ensuring that features computed at serving time (streaming, low-latency) match features computed at training time (usually batch, over historical data). If the batch feature pipeline and the streaming feature pipeline implement the same logic independently (e.g., 'transactions in last 10 minutes' coded once in Spark SQL and once in Flink), subtle differences in windowing semantics, timezone handling, or null-handling cause training-serving skew — a model that performs well offline but degrades in production because the features it sees in production don't statistically match what it was trained on.

Feature stores exist to solve exactly this

Modern feature store architectures (Feast, Tecton, Databricks Feature Store) address training-serving skew by defining feature logic once and compiling it to both an offline (batch, for training) and online (low-latency, for serving) execution path, guaranteeing point-in-time correctness — training examples only see features as they existed at that historical moment, not features computed with 'future' information.

Exactly-once is harder than it sounds

Network retries and consumer failures mean a naive streaming pipeline can double-count events. True exactly-once semantics require idempotent writes (keyed by event ID) or transactional sinks coordinated with the broker's offset commits — without this, aggregate features like 'spend in the last hour' silently inflate under retries, degrading model accuracy in ways that are hard to detect from monitoring alone.

What's Next?

Streaming pipelines are one half of the equation; the other is how real-time ('speed layer') and batch ('historical layer') processing are architecturally combined so a system gets both low latency and eventual accuracy.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →