Success Metrics
A while ago, during my early days of ‘formal’ product management career, I was discussing a new feature (a new type of job listing) with a senior business leader. Even before implementing the same, he suggested that we should create a dashboard that tracks the number of listings created per day.
I asked with genuine curiosity: “Why do we need the dashboard ?” The answer was something similar to:
“If you can’t measure it, you can’t improve it”
I measured it but it didn’t improve. I realised that if you keep measuring something it will not improve on its own.
But it kept us on our toes. We then had to figure out lot of creative growth tactics and feature improvements to improve the metric.
There are arguments against this approach. That not everything important can be measured like happiness, team bonding, aesthetics, user experience and so on.
I’d like to believe that in product management, especially in digital products, success of most initiatives can be measured.
Now, how do we arrive at the right metric(s) to measure success ?
From the book Shipping Greatness by Chris Vander Mey
There’s a story, possibly apocryphal, that tells how Frito-Lay came up with one metric by which it could run its business. Frito-Lay stocks store shelves, taking up critical inventory space. Ideally, it will take up exactly the amount of space on a shelf that it needs — too much, and its products get returned. Too little, and it misses out on sales.
Frito-Lay solved this problem by measuring “stales,” the count of products that are returned at each restock event because the product is out of date. Frito-Lay wants the number of stales to equal precisely one. Taking a single bag of potato chips as a chargeback may seem a crime to you, but as a measurement cost it is very small. If the stales count is greater than one, the suppliers decrease the stock levels. If there are no stales, they increase.
A great metric:
- Is inexpensive to measure
- Can be measured reliably and repeatedly
- Is measured frequently, ideally in real time
- Enables your team to make smart changes (actionable)
- Reflects the customer experience
CONVERSION FUNNELS
For consumer internet apps, web-analytics experts recommend measuring macro-conversion (or conversions that directly impact revenue) and micro-conversions (smaller engagements that eventually lead to macro-conversions).
Macro-conversions are usually broken down into multiple steps; each step completion being a micro-conversion. More importantly, each step completion can be improved independently by working on different aspects of the process.
For Example:
- Success of a mailer campaign is split into open rates, click through rates & further steps if any.
- Conversion of a classified site can be split into visitors landing on the home page, percentage of them initiating a search, clicking on a search result & viewing the listing details page and finally converting with the Call-To-Action
Further, the funnel is sliced and analysed through various segmentation schemes like source of traffic, geography, visitor cohort by acquisition date and so on.
AARRR Framework
Dave McLure in this slideshare talks about tracking the customer life cycle with the AARRR framework for growing a product.
- Acquisition: users come to the site from various channels
- Activation: users start using the product
- Retention: users come back, visit multiple times
- Referral: users like the product enough to refer others
- Revenue: users conduct some monetisation behaviour
Google HEART Framework
Google in this research paper describes the HEART framework to measure user-centered metrics for web apps.
Happiness: Metrics that are attitudinal in nature. These relate to subjective aspects of user experience, like satisfaction, visual appeal, likelihood to recommend, and perceived ease of use. Measurable by a well-designed survey.
Engagement: User’s level of involvement with a product; in the metrics context, the term is normally used to refer to behavioural proxies such as the frequency, intensity, or depth of interaction over some time period.
Adoption & Retention: Adoption metrics track how many new users start using a product during a given time period (for example, the number of accounts created in the last seven days), and Retention metrics track how many of the users from a given time period are still present in some later time period (for example, the percentage of seven-day active users in a given week who are still seven-day active three months later)
Task Success: Task Success category encompasses several traditional behavioural metrics of user experience, such as efficiency (e.g. time to complete a task), effectiveness (e.g. percent of tasks completed), and error rate. Measured by usability or benchmarking study or UI event tracking.
GOALS — SIGNALS — METRICS
Goals: The first step is identifying the goals of the product or feature, especially in terms of user experience. Use the HEART framework to prompt articulation of goals (e.g. is it more important to attract new users, or to encourage existing users to become more engaged?)
Signals: How success or failure in the goals might manifest itself in user behaviour or attitudes
Metrics: How these signals can be translated into specific metrics, suitable for tracking over time on a dashboard
Reading the paper is highly recommended
All these frameworks help us define metrics to measure success, and force us to think of ideas to improve on each dimension.
Outcome-Driven-Innovation
I recently went through this book: What Customers Want: Using Outcome-Driven Innovation to Create Breakthrough Products and Services by Anthony Ulwick
The author describes a metric-driven process to do market research and create products that match the needs of targeted segments in the market. It is built around this theory that people buy products and services to get jobs done. As people complete these jobs, they have certain measurable outcomes that they are attempting to achieve, within certain constraints.
For example house painters had outcomes like:
- Minimise the amount of paint wasted due to over-purchase
- Minimise the time it takes to repair the surface defects
Qualitative and quantitative research is conducted to identify the outcomes, their importance and current satisfaction levels in the product. This data is further used in segmentation, targeting, positioning, prioritising and focused brainstorming.
Once we make an improvement in the product, success can be measured as change in satisfaction levels of the targeted outcome.
The book gave me a lot of ‘aha moments’ and is highly recommended.
Finally, let me reiterate that
- It is important to define and measure success metrics.
- It is possible to define success metrics for most initiatives.
- Right success metrics can help us focus our energies to generate, identify and test the right ideas.
You may also like:
On what goes into a product roadmap and the perils of a static perspective: Zooming out, drilling down and changing hats
Managing machine learning based products and model evaluation metrics: How to evaluate your model?
A brief note on causal inference for product managers
How to add machine learning to your product?
On tracking and measure product KPI-s: Success Metrics
On learning continuously: About Curiosity, Learning and Eigenvectors
