Vanity Metrics vs Outcome Metrics: What Product Leaders Should Measure

A product review opens with a slide: two million downloads, forty thousand new signups this quarter, a features-shipped count that has grown every month for a year. The room nods. Someone says “great quarter.” Then a quieter question gets asked, almost as an afterthought: how many of those forty thousand new signups are still using the product, and are they actually better off because of it? The room goes still, because nobody prepared that slide. This is the moment vanity metrics vs outcome metrics stops being an academic distinction and starts being the difference between a team that knows what it accomplished and a team that only knows what it counted.

What Makes a Metric a Vanity Metric

A vanity metric is any number that goes up easily, looks good on a slide, and does not, by itself, tell you whether a customer’s situation improved. Downloads, total signups, page views, total registered users, and features shipped are the classic examples. None of them are fake. They are real counts of real events. The problem is not that they are wrong, it is that they are incomplete in a specific and dangerous way: they measure exposure or activity, not change.

Outcome metrics measure something different: a behavior that changed, a problem that got resolved, a measurable result in the customer’s life or work. Joshua Seiden’s framing in Outcomes Over Output is useful here: an outcome is a change in customer behavior that drives results for the business, not an event that happened to the business itself. Output is what a team creates, an app, a signup flow, a feature, a report. Outcome is what becomes better because that output existed. A download is output reaching a device. It is not evidence that anything got better.

A complaint is not automatically a high-priority product problem, and a download is not automatically a sign of value delivered. Both require a second question before they mean anything.

Why Vanity Metrics Are So Easy to Report

Vanity metrics survive in leadership decks for the same three reasons they always have. They are visible: a download count or a signup number is a single, unambiguous figure that needs no interpretation. They travel well upward through management, because “we hit two million downloads” requires no follow-up explanation the way “trust increased” does. And they always move in a comfortable direction, because totals accumulate. A cumulative signup count almost never goes down, which makes it a pleasant number to put in front of a board even when the business underneath it is stalling.

None of that makes the number a poor thing to track internally. Total users is a legitimate operational fact. The trouble starts when that operational fact quietly gets promoted into evidence that a product is succeeding, which is a claim it was never built to support.

Pairing the Vanity Metric With the Outcome Metric

The fix is rarely “stop counting the vanity metric.” It is “stop treating it as the answer, and pair it with the metric that actually answers the question.” A few concrete, illustrative pairings make this tangible.

Downloads → Activation and repeat use. A hypothetical medication-reminder app reports 500,000 downloads in its first year. That number says people found the app store listing and tapped install. It says nothing about whether anyone changed how they take medication. The outcome metric sitting beside it might be: percentage of downloaders who set up at least one reminder within 48 hours, and percentage still logging a dose at 30 days. Downloads measure reach. Activation and retention measure whether the output did the job it was built for.

Total signups → Problem resolution rate. A benefits-navigation platform, again hypothetically, signs up forty thousand new members in a quarter. The outcome metric that belongs next to it is the share of those members who actually found what they needed, resolved a claims question, or completed an enrollment task without contacting support. A signup is consent to try. It is not proof of a problem solved.

Page views → Task completion. A help center redesign drives page views up sharply after launch. That is easy to celebrate and easy to misread. The relevant outcome metric is whether visitors who land on an article actually complete the task the article was meant to support, resetting a password, understanding a bill, filing a claim, without escalating to a support ticket afterward. Traffic to an article is not the same as the article having done its job.

Features shipped → Adoption and behavior change. A team ships eighteen features in a quarter and reports the count with pride. Pendo’s 2019 Feature Adoption Report, based on an analysis of roughly 615 software subscriptions, found that around 80 percent of features in the products it studied were rarely or never used. That is one vendor’s dataset, not a universal law, but it is a strong caution against treating a shipped-features count as a proxy for value. The outcome metric that matters is how many of those eighteen features are used regularly by the core customer segment six weeks after launch, and whether any measurable customer behavior changed as a result.

Total registered users → Active problem-solving. A dashboard product reports 100,000 registered accounts. Many products carry years of accounts that logged in once and never returned. The outcome metric worth reporting instead is weekly or monthly active users who complete a defined, valuable action, not simply users who exist in a database.

In each pairing, the vanity metric is not deleted. It still has a legitimate operational use, as a top-of-funnel or reach indicator. What changes is which number gets to answer the question “is this working,” and that job goes to the outcome metric every time.

The Trap: When the Headline Number Rises and the Outcome Doesn’t

The most dangerous version of this problem is not a team that only tracks vanity metrics. It is a team that tracks an outcome-shaped number, watches it rise, and never checks what is happening underneath it. A metric can go up while the real customer outcome stays flat or actively worsens, and the surface number will not tell you.

Consider a hypothetical customer support chatbot rolled out to deflect tickets. Deflection rate, the percentage of conversations resolved without reaching a human agent, climbs from 30 percent to 65 percent within two months. On a slide, that looks like a clear win. But deflection rate only measures whether a human agent got involved, not whether the customer’s problem actually got solved. If the underlying issue is that the bot is closing conversations by giving customers an unclear or wrong answer and customers simply give up rather than escalate, deflection rate rises while customer problem resolution falls. The team would be optimizing for a number that goes up precisely because the outcome is getting worse.

This is the core trap behind vanity metrics vs outcome metrics: even a metric with “outcome” in its description can behave like a vanity metric if it can be inflated by something other than the change it claims to represent. A successful-looking number should never be trusted in isolation.

Guardrails Catch What the Headline Number Hides

This is exactly the failure mode that guardrail metrics exist to catch. A measurement approach built around a single Primary Outcome Metric plus a small set of guardrails is designed so that a rising headline number cannot hide a worsening customer situation without tripping something else.

For the chatbot example, sensible guardrails would include repeat-contact rate within 24 hours of a “deflected” conversation, customer-reported satisfaction on those conversations, and escalation requests explicitly overridden or ignored by the bot. If deflection rate climbs to 65 percent while repeat-contact rate within a day also climbs, that is not a support-efficiency win, it is a customer being turned away twice for the same problem, once by the bot and once by the metric that failed to flag it. A successful outcome should not hide an unacceptable side effect.

The same logic applies to the earlier pairings. If activation rate for the medication app rises but a guardrail tracking reported dosing errors also rises, the metric is not telling the full story. If problem-resolution rate for the benefits platform improves but complaint volume about being misdirected also grows, something is off underneath a number that looks fine on its own. An assumption written in a product document does not become a fact just because a chart is trending upward, and a metric does not become trustworthy just because it is moving in the intended direction. It becomes trustworthy when a guardrail has had the chance to catch it and hasn’t.

Building this pairing into a team’s regular reporting rhythm does not require new tooling or a quarterly overhaul. It requires the discipline, at the point a metric is chosen, to also name what would have to be watched to know if that metric is lying. Teams that work through the framework’s measurement stage as part of planning an initiative, rather than after it launches, tend to find this pairing far easier to define up front than to retrofit onto a dashboard that already exists. Organizations building this discipline for the first time sometimes bring in outside structured support simply to get the first few metric-guardrail pairs right before the habit becomes internal.

Key Takeaway

Vanity metrics like downloads, signups, page views, and features shipped are not false numbers, but they answer “did activity happen,” not “did a customer’s situation improve.” Every vanity metric worth reporting should sit next to an outcome metric that measures behavior change or problem resolution, and every outcome metric worth trusting needs a guardrail that would catch it if the number rose for the wrong reason. A metric that cannot be checked against a guardrail is a number waiting to mislead someone.