A Bias Audit Found a Real Problem We Almost Shipped

A resume screening model a client was building passed every aggregate accuracy benchmark comfortably before we ran a dedicated bias audit, splitting performance by demographic group rather than looking at the overall number alone. The audit found a disparity in false rejection rates between groups large enough that shipping without addressing it would have been both an ethical problem and a real legal exposure for the client.

How the disparity hid behind a good aggregate number

The model's overall accuracy was strong because the majority group in the training data was large enough to dominate the aggregate metric. The false rejection rate for a smaller demographic group in the data was meaningfully higher — qualified candidates from that group were being screened out at a noticeably elevated rate compared to equally qualified candidates from the majority group. None of that showed up in the overall accuracy number, which stayed flat and reassuring the entire time.

A fairness problem can hide behind a healthy aggregate accuracy score for as long as nobody asks the model to break its own performance down by group.

Where the disparity actually came from

Tracing it back, the root cause was a proxy feature — a résumé formatting pattern correlated with where candidates had gone to school, which was in turn correlated with the demographic group showing the disparity. The model hadn't been given demographic data directly and didn't need it; it had found a proxy that carried the same signal, which is exactly the trap that makes bias audits necessary even on models that were deliberately built without protected-class features.

• Aggregate accuracy metrics can mask subgroup disparities entirely — a demographic-split evaluation needs to run as a standard part of model validation.
• Removing a protected attribute from the feature set doesn't prevent the model from learning a correlated proxy for it.
• The audit needs to happen before launch, when the fix is a retrain, not after launch, when it's a legal and reputational incident.
• Fixing the disparity required identifying and addressing the specific proxy feature, not just reweighting the loss function generically.

We retrained with the proxy feature removed and added a standing fairness evaluation to the model's validation pipeline going forward, so this check runs automatically on every future retrain rather than depending on someone remembering to ask for it. The client avoided shipping something that would have quietly screened out qualified candidates at a disproportionate rate, and the only reason is that someone insisted on looking past the aggregate number before launch.