Responsible AI Product Management

A model can achieve 95% overall accuracy while performing at 99% accuracy for one group and 70% for another, and the aggregate number alone would never reveal this. This is the same base-rate and aggregation risk introduced in Lesson 65's discussion of accuracy on imbalanced classes, now applied specifically to demographic or protected-group disparity rather than class imbalance. The arithmetic reason this happens is straightforward once stated: if one group makes up 90% of the training data and the model performs well on that majority group, the aggregate accuracy figure will be dominated by that majority group's performance almost regardless of how poorly the model does on the remaining 10%. A team that only ever looks at the single headline accuracy number has, in effect, built a measurement system that is structurally blind to exactly the kind of harm a smaller or underrepresented group is most likely to experience — which is precisely why subgroup measurement cannot be treated as an optional, nice-to-have addition to standard model evaluation.

This lesson introduces the Fairness Audit Loop:

The loop's discipline is that it never terminates — remediation feeds back into re-measurement, since a fix applied once can itself introduce a new disparity elsewhere, or can decay as data and usage patterns shift over time.

A superficial fix — adjusting a model's output thresholds differently by group after the fact — can mask rather than resolve the underlying issue, and can itself introduce new legal and ethical complications. Genuine remediation investigates whether the disparity traces to biased training data, a proxy variable correlated with a protected characteristic, or a genuine difference in the underlying task that requires a different solution entirely.

Responsible AI deployment requires that affected individuals can understand, at some level, why a decision was made, and have a genuine path to contest it — directly connecting to the Escalation Staircase's appeals requirement from Lesson 67, now applied specifically to AI-driven decisions.

A specific complication that trips up even well-intentioned teams: there is no single, universally agreed mathematical definition of fairness, and several reasonable-sounding definitions can be mutually incompatible with each other for the same decision. A model can achieve equal approval rates across groups (demographic parity) while still producing unequal error rates within those groups (unequal false-positive or false-negative rates), and it is mathematically impossible, in most realistic cases, to satisfy both definitions simultaneously if the underlying base rates differ across groups. This means a PM cannot simply instruct a team to "make the model fair" and expect a single unambiguous target — the team must first make an explicit, documented choice about which fairness definition is appropriate for the specific decision at hand (a lending decision, a hiring screen, a content-moderation call), and that choice itself deserves the same scrutiny and stakeholder input as any other significant product decision, since different definitions can be more or less appropriate depending on the real-world consequences of false positives versus false negatives for the specific population affected.