Lesson 65: Working with Data Science & ML Teams

Lesson 64 established that metric trust, not data volume, is the actual constraint on data-informed decision-making — and introduced the Metric Provenance Chain as a way to verify that a number deserves to inform a decision. This lesson extends that discipline into a specific and increasingly common working relationship: the one between a PM and a data science or machine learning team building a model whose output will itself become an input to product decisions.

This relationship fails in a very particular and recurring way. A PM, trained to think in terms of user problems and business outcomes, hands a data scientist a goal like "predict which customers will churn" and treats the resulting model as a black box that will simply be correct. A data scientist, trained to optimize a technical objective, builds a model that scores well against a chosen metric — precision, recall, AUC — without necessarily understanding, or being told, what the cost of a wrong prediction actually is in the product experience the PM is responsible for. Both sides can do excellent, technically sound work, and the collaboration can still fail, because no one made explicit who owns which part of the decision chain from raw prediction to shipped product behavior.

This lesson introduces the Ownership Zones Model, a way of dividing the PM/data-science relationship into distinct zones of responsibility, so that the most common and expensive failure in this collaboration — a model that is technically excellent but wrong for the product's actual needs — becomes visible and preventable before it ships.

Learning Objectives

1
Explain why a technically excellent model can still be the wrong model for a product's actual needs.
2
Apply the Ownership Zones Model to clarify which decisions belong to the PM, which belong to data science, and which are shared.
3
Distinguish precision-oriented and recall-oriented error costs and explain why they are a product decision, not a purely technical one.
4
Identify what a PM must supply to a data science team before model development begins.
5
Evaluate a proposed model deployment for whether its ownership zones and error costs were made explicit before shipping.

This lesson assumes the Metric Provenance Chain and Trusted Metric concepts from Lesson 64, since a model's output is itself a metric that must earn trust before informing decisions, and the A/B testing rigor from Lesson 45, since most model deployments are validated using the same experimental discipline.

Why a Technically Excellent Model Can Still Be Wrong

A model can score well against the metric it was optimized for and still fail the product it's meant to serve, because the technical metric a data scientist optimizes (accuracy, precision, recall, AUC) is a proxy for a business outcome the PM actually cares about, and proxies can diverge from the real target in ways invisible to the model-building process itself. A churn-prediction model with 90% accuracy sounds impressive until you learn that only 5% of customers actually churn in a given period, meaning a model that simply predicts "no one churns" would already be 95% accurate — a direct echo of Goodhart's Law from Lesson 41, now applied to a model's own training objective rather than a company's dashboard metric.

The Ownership Zones Model

This lesson introduces the Ownership Zones Model, dividing the path from a business problem to a shipped model-driven product decision into four zones:

Zone 1, Problem Framing, is owned by the PM: defining what business outcome the model should serve, what a correct versus incorrect prediction actually costs in the product experience, and what threshold of performance would make the model worth deploying at all. Zone 2, Model Development, is owned by data science: choosing the modeling approach, features, and technical optimization target, informed by the framing PM supplied in Zone 1. Zone 3, Output Interpretation, is genuinely shared: understanding what the model's output actually means in practice (a "churn probability of 0.7" is not the same as "this customer will churn," and translating between the two requires both statistical literacy and product judgment). Zone 4, Product Decision and Deployment, returns to the PM: deciding what the product actually does with a given model output — does a high churn-risk score trigger an automated retention email, a customer-success outreach, a discount offer, or nothing at all — a decision that depends on business context data science does not own.

The most common and costly failure in PM/data-science collaboration is a breakdown at the Zone 1/Zone 2 boundary: a PM who hands over a vague goal without specifying error costs, and a data scientist who, in the absence of that specification, optimizes for a generic technical metric that may not reflect what the product actually needs.

Precision, Recall, and Error Cost as a Product Decision

Precision measures, of everything the model flagged as positive (for example, "will churn"), what fraction actually was positive. Recall measures, of everything that actually was positive, what fraction the model successfully flagged. These two properties trade off against each other, and which one to prioritize is fundamentally a product decision about the relative cost of two different kinds of errors, not a purely technical one.

If the cost of a false positive is low (a slightly unnecessary retention email) and the cost of a false negative is high (losing a valuable customer with no intervention attempted), the PM should direct data science toward a recall-oriented model, accepting more false positives to catch more true churners. If the reverse is true — false positives are expensive (an aggressive, unwanted discount offer that trains customers to expect discounts) and false negatives are comparatively cheap — a precision-oriented model is the better fit. Data science cannot make this trade-off correctly without the PM supplying the business cost information that only the PM, as the owner of Zone 1, actually has.

What a PM Must Supply Before Model Development Begins

Before Zone 2 (Model Development) can proceed responsibly, the PM should supply, at minimum: the specific business decision the model's output will inform, the relative cost of false positives versus false negatives in that decision context, any known segments where errors would be especially costly or especially tolerable, and the minimum performance threshold below which the model should not be deployed at all, regardless of how it compares to alternatives. Absent this input, data science teams are left to infer these judgments themselves, and reasonable technical assumptions can diverge sharply from actual business reality.

Common Mistakes to Avoid

Handing data science a vague goal without specifying error costs

"Build a model to predict churn" leaves the precision/recall trade-off, which is a product decision, to be made implicitly and invisibly by whoever builds the model.

Treating a model's technical metric as automatically equivalent to product success

A model with high accuracy or AUC can still be the wrong choice if that metric doesn't reflect the actual cost structure of the product decision it feeds.

Assuming a model's output is a fact rather than a probabilistic estimate

Treating a "churn probability of 0.7" as equivalent to "this customer will churn" collapses Zone 3's necessary interpretation step and leads to overconfident product decisions.

Deploying a model without a defined Zone 4 action plan

Building a model before deciding what the product will actually do with its output wastes data science effort and often results in a model that sits unused because no clear decision was ever attached to it.

Never revisiting the ownership zones after deployment

Business context, error costs, and the product decisions attached to a model's output can all shift over time, and failing to periodically revisit Zone 1 framing can leave a once-appropriate model quietly misaligned with current business needs.

Working with Data Science & ML Teams