An AI Ethics Review Changed a Feature's Scope Before Launch

We had an AI feature ready for launch — a model that scored customer support interactions for agent coaching purposes — and the ethics review was, honestly, scheduled more as a box-checking formality than an expectation of real findings. It ended up reshaping the feature's scope before launch in a way that made the shipped version better, not just safer.

What the review actually surfaced

The original design scored agents on a composite metric that blended several signals, including customer sentiment inferred from the conversation. The review raised a concrete concern: sentiment inference from text is noisier for non-native speakers and for customers communicating in a second language, and folding that noisy signal into an agent's performance score risked systematically penalizing agents who happened to handle more of those conversations, for reasons that had nothing to do with their actual performance.

A model doesn't need to be malicious to produce an unfair outcome. It just needs a noisy signal and nobody checking whether the noise is evenly distributed.

What changed in the shipped feature

We pulled sentiment inference out of the composite score entirely and shipped it as a separate, clearly-labeled signal for the agent's own awareness rather than folding it into a number used for performance evaluation. The composite score that remained, built from things like resolution time and follow-up rate, was arguably a cleaner, more defensible metric on its own merits — not just a safer one.

• Noisy signals get surfaced separately, not folded into composite scores used for consequential decisions.
• Any signal with known accuracy gaps across subgroups gets flagged before it touches an evaluation metric, not after a complaint.
• The review happened before launch, when the fix was a design change, not after launch, when it would have been a retraction.
• A narrower, more defensible feature shipped faster than the original broader one would have, once the rework was scoped.

I went into that review expecting a rubber stamp and came out with a materially different, better feature. The lesson wasn't "ethics reviews catch bad actors" — it was that a structured check for exactly this kind of subtle, well-intentioned mistake catches things a team moving fast toward a launch date genuinely won't catch on their own.