Data Engineering for Machine Learning

Garbage in, garbage out is an engineering problem

The phrase 'garbage in, garbage out' gets used as a vague warning about data quality, but for ML systems it's precise and mechanical: a model is a function fit to whatever distribution of examples it's shown, so any systematic flaw in that distribution — missing segments, mislabeled examples, leaked future information, inconsistent units — becomes a systematic flaw in the model's behavior, encoded permanently into its weights until retrained.

This reframes data quality from a one-time cleanup task into infrastructure: pipelines that validate schemas, detect anomalies, version datasets, and guarantee the exact same feature computation at training and serving time. Data engineering for ML is the discipline of building that infrastructure so data quality is enforced continuously rather than checked manually before each training run.

ETL vs. ELT, and why ML tilts toward ELT

ETL (Extract, Transform, Load) transforms data before it lands in a destination store — you decide the shape of the data up front. ELT (Extract, Load, Transform) loads raw data first and transforms it afterward, usually inside a data warehouse or lake, using the destination's own compute.

ML workloads tilt toward ELT because feature requirements change constantly during experimentation — a data scientist wants to try a new aggregation window or a new derived feature weekly, and re-running a rigid upstream ETL pipeline for every experiment is slow. Keeping raw data intact and transforming it downstream, on demand, lets feature engineering iterate without re-architecting the ingestion pipeline. The tradeoff is cost and governance: raw data stores grow large, and without discipline, every team reinvents its own slightly different version of the same feature.

ETL vs. ELT for ML pipelines

Feature stores: solving train/serve skew at the infrastructure layer

A feature store is a system that computes, stores, and serves features for both training and inference from one shared definition, instead of letting each pipeline implement feature logic independently. It typically has two sides: an offline store (large-scale batch computation, feeding training jobs with historical feature values) and an online store (a low-latency key-value store, usually Redis or DynamoDB-backed, serving the same features to a live model at inference time).

The point of a feature store isn't just convenience — it's structural prevention of train/serve skew. If the training job and the serving path both call the same feature definition (avg_purchase_30d, computed once, served two ways), there is no code path where the two can silently diverge, which is the exact failure mode covered in ML System Design Fundamentals.

Data versioning: treating datasets like code

Code is versioned by default in every modern engineering org; datasets, historically, are not — and that asymmetry is where a lot of ML incidents originate. Data versioning means every training run is tied to an immutable, identifiable snapshot of the exact data used, so that a model's behavior can always be traced back to what it was trained on, and a 'bad' dataset change can be rolled back the same way a bad commit can.

Tools like DVC, LakeFS, or Delta Lake's time-travel features exist specifically to make large, frequently changing datasets versionable without duplicating the storage cost of every version. Without this, debugging 'why did the new model regress' often turns into reconstructing, from memory or scattered logs, what the training data even looked like three weeks ago.

Validation as a pipeline stage, not a one-off check

Production data pipelines should run automated validation on every batch: schema checks (are the expected columns and types present?), statistical checks (has the distribution of a key feature shifted sharply since yesterday?), and referential checks (do foreign keys actually resolve?). Tools like Great Expectations, TensorFlow Data Validation, or Deequ exist to make these checks declarative and automatic rather than something an engineer remembers to eyeball.

The goal is to catch 'garbage in' at the pipeline boundary, before it reaches a training job or a feature store — because once bad data has been baked into a trained model, the fix is an expensive retrain rather than a cheap pipeline patch.

What's next

Well-engineered data gets a model trained correctly — but a correctly trained model is often too large or too slow to run where it needs to run. The next lesson, Model Compression: Quantization, Pruning & Distillation, covers the techniques used to shrink a model for production without destroying the accuracy this data pipeline worked so hard to earn.