Pitfall #4 - measuring the wrong feature-level metrics
Efficiently diagnose underperforming features.
đ Howdy, Erik here. Welcome to the 4th edition of Early Product Pitfalls. Each week I bring founding teams at startups âł two minutes on a not-so-obvious pitfall when building new things.
đłThe pit
Launching features with the intention of measuring a smattering of engagement metrics to eyeball success or failure.
đ¤ What you might sound like
Weâre getting data back on what we shipped last week. Some engagement metrics are good. Some are ok. Some arenât great. What should we do next?
It doesnât make sense to measure anything yet. We wonât have statistical significance for a while given the size of our user base.
đ You, knowing better âŚ
You donât consider anything fully shipped until it is being actively measured with behavioral analytics. You recognize that each feature targets a subset of your user base. You have specified that target qualitatively and can filter for them in your data.
For everything you ship, you have identified three metrics that each answer a critical question related to feature success:
Adoption - what percent of your featureâs target audience used your feature at least once?
Retention - what percent of those who adopted your feature continued to use it over time?
Satisfaction - what percent of those who adopted your feature were satisfied with how it worked?
When adoption scores low you investigate discoverability and question how your feature is communicating its value.
When retention scores low you investigate the value of what the feature does and its natural frequency.
When satisfaction scores low you improve its usability. Because of this focus, you are able to easily hypothesize how you might improve your features after launch.
đ Example
Slack is an instant messaging platform for work teams.
Huddles
In 2021, Slack launched Huddlesâa way for users to quickly connect on a voice or video call.
Measuring Success
Imagine that we plan to launch Huddles to a subset of our user base that represent the featureâs target audience. In order to do that, weâll need to define this audience.
Huddles were designed for team member collaboration. The target audience weâll want to measure might be described as users who:
Were included in the initial rollout
Belong to company Slack workspaces vs. large communities
Are active post release
Here is how we might decide to measure adoption, retention and satisfaction âŚ
Adoption: % of our target that places one call that connects and lasts at least 30 seconds
Retention: % of adopted users who continue to place one weekly call over time
Satisfaction: average ease of use score that is measured on a likert scale ranging from very difficult to very easy
đŚ Light for the dark
âYesterday I was clever, so I wanted to change the world. Today I am wise, so I am changing myself.â
Rumi
