The Fairness-Accuracy Tradeoff in a Credit Scoring Model Project

A lender client wanted a credit risk model refreshed. Early in feature selection, ZIP code showed up as one of the top five predictors by importance. It also happens to correlate heavily with race in the US, for reasons that have nothing to do with creditworthiness and everything to do with redlining history. The model wasn't doing anything exotic — it was just very good at finding a legal proxy for a protected attribute, because that proxy really does carry predictive signal in the historical data.

The honest version of the tradeoff

Removing ZIP code outright dropped model accuracy by about two points of AUC. That's not nothing on a portfolio this size — it translates to real dollars in default rates either way you cut it. The team that wanted to keep it wasn't being careless; they were pointing at a real number and a real business cost, and pretending that cost doesn't exist makes for a worse conversation, not a better decision.

Pretending a fairness tradeoff costs nothing doesn't make the decision easier, it just makes it dishonest.

What we actually did

We didn't just drop the feature and declare victory. We ran a disparate impact analysis with and without it, quantified exactly how much the proxy was driving approval rate gaps across demographic groups, and brought both the accuracy cost and the fairness cost to the table as two numbers instead of one side pretending the other didn't matter. The decision — removing ZIP-derived features and accepting the accuracy hit — got made by compliance and the business together, with the tradeoff written down, not buried in a model card nobody reads.

• Measure disparate impact before and after removing a suspect feature, not just overall accuracy.
• Quantify the dollar cost of the accuracy drop so the tradeoff is concrete, not abstract.
• Put the decision in front of people with actual authority over the risk, not just the modeling team.
• Document the reasoning, because regulators and future engineers will both eventually ask why.

The version of this story that makes a good conference talk is 'we fixed bias and nothing was lost.' The honest version is we lost something measurable and decided it was worth it, with the people who actually bear that cost in the room when we decided.