Event Streaming with Kafka for Marketing
Why marketing data eventually outgrows batch ETL
Batch ETL (scheduled jobs pulling and transforming data every hour or every night) is simple, cheap, and sufficient for most reporting needs — but it has a hard latency floor that no amount of optimization within the batch model removes. Once a marketing use case genuinely requires reacting to an event within seconds or minutes rather than hours (real-time personalization, fraud/bot detection that needs to block before the next page load, triggering an abandoned-cart email within minutes of the abandonment), batch scheduling becomes the bottleneck, and the architecture needs to shift to event streaming — processing events continuously as they happen rather than in periodic scheduled chunks.
Kafka's role: a durable, replayable event log
Kafka is a distributed event streaming platform organized around topics (named event streams, e.g. page_views, purchases, email_opens) that producers write events to and consumers read from, independently and at their own pace. The property that makes Kafka specifically valuable over a simpler message queue: it retains events for a configurable retention period (not just until a consumer reads them), which means multiple independent consumers can read the same event stream for entirely different purposes — one consumer feeding a real-time personalization engine, another writing to the data warehouse for later analysis, another triggering fraud-detection scoring — without any of them interfering with each other or needing the producer to know about them.
Batch ETL vs. Kafka-based event streaming
| Dimension | Batch ETL | Kafka streaming |
|---|---|---|
| Latency floor | Set by schedule interval (minutes to hours) | Seconds, sometimes sub-second |
| Multiple independent consumers of the same data | Requires re-running the batch job or duplicating pipelines | Native — any number of consumers can read the same topic independently |
| Operational complexity | Lower — cron/Airflow and a warehouse | Higher — cluster management, partition/consumer group tuning, schema evolution |
| Replay/reprocessing | Re-run the batch job against historical source data | Native — consumers can re-read from any retained offset in the topic |
| Right fit for most reporting dashboards | Yes — reporting rarely needs sub-hour latency | Overkill for pure reporting; justified when real-time action is the actual requirement |
Don't adopt Kafka because real-time sounds impressive — adopt it because a specific use case needs sub-minute latency
// Simplified consumer pattern: a real-time personalization service
// reading from a Kafka topic independently of the warehouse-bound consumer.
import { Kafka } from "kafkajs";
const kafka = new Kafka({ clientId: "personalization-engine", brokers: ["broker1:9092"] });
const consumer = kafka.consumer({ groupId: "personalization-group" });
async function run() {
await consumer.connect();
await consumer.subscribe({ topic: "page_views", fromBeginning: false });
await consumer.run({
eachMessage: async ({ message }) => {
const event = JSON.parse(message.value?.toString() ?? "{}");
// Update an in-memory or low-latency store (e.g. Redis) with the
// user's latest browsing signal, read by the personalization API
// the website queries on next page render.
await updateUserSignalStore(event.user_id, event.page_category);
},
});
}
run();Handling late and out-of-order events
Streaming introduces a problem batch processing sidesteps almost entirely: events can arrive out of order (a mobile client buffering events offline and flushing them later) or genuinely late relative to other events in the same user session. Consumers that assume strict chronological order will misbehave on this traffic — a session-reconstruction consumer needs explicit logic for watermarking (defining how long to wait for late-arriving events before finalizing a session window) and deduplication (handling the same event being redelivered after a consumer restart, which Kafka's at-least-once delivery semantics make a real and expected occurrence, not an edge case).
Schema evolution is the operational tax nobody budgets for upfront
What's Next?
Event streaming is the plumbing that makes truly real-time marketing capabilities possible — the next question is what you actually build on top of that plumbing, starting with the real-time personalization infrastructure that consumes these streams to make moment-by-moment decisions.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.