Last year, our team got an urgent message. The experience on one of our key products was receiving a slew of negative user feedback. Could we fix it? Slated to be replaced, the product was largely unowned at the time. As our team started digging into the root cause, we realized there weren’t any silver bullets. What followed was a year of small wins and hard fought lessons on what it means to define and improve quality.
The quality of our products influence the emotions they evoke. As product builders, we hope that people describe our products as delightful, simple, beautiful, and thoughtful. But quality comes at a cost. We trade off between perfecting the product, against the opportunity cost of adding more value. In a market where the problems to solve are ill-defined, perfection is not always the clear winner.
How do we balance our pursuit of quality with shipping faster?
- Align on the quality bar for launch
- Measure, and measure again
- Chisel the best products to refine them
- Remove the excess
- Sharpen our saw
- (Bonus) Create incentives that reward improvements at each of these steps
1. Align on the quality bar for launch
Quality is a fuzzy word. For quality to factor in our decision making process, we need a shared vocabulary and understanding of what is good enough to launch. By building a process around how we can trade off between learning faster and creating a great experience, we empower teams to align on what is considered shippable.
I first came across this framework from @jeffseibert at Fabric, and it’s served me well to drive alignment on what we are comfortable shipping.
When issues are raised as quality blockers, they get elevated and discussed in the same way as other launch blockers.
2. Measure and measure again
Our products are built on layers of complexity. When things aren’t working well, there often isn’t just one simple reason. There are two approaches that we have found to work that need to happen in parallel,
- Build resilient systems that can recover from failures
- Build a deep understanding of where in the process things are broken.
The best investment we have made here has been developing rigorous instrumentation and working with Data Scientists and User Experience Researchers to build a shared understanding of what good looks like.
For our product, everyone could point to it and call out that things were not working, but to figure out why, we had to fix instrumentation every step of the way, and even checking whether the logging itself was correct. Ultimately, the biggest win we had was just adding a simple retry logic so that the product could correct for temporary failure cases.
3. Chisel the best products to refine them
The glamorous first launch is followed by a long period of iterative refinement. While each change is not remarkable, the compounding effect over time is undeniable. Perhaps the most consistent example of this are Apple computers where every generation improves on thickness and display.
For refining to work, we need to lock down a few areas that are invariants to a good experience. And then consistently raise the bar on those.
4. Remove the excess
Every feature requires maintenance and an opportunity cost of something else that could take its place. Removing old features in a paid product is tricky with concerns that some people may have only bought our product for that reason.
As we start cleaning house, we will need to confront the question of what features we should remove?
While low adoption should trigger a review, naive approaches using a small set of metrics to make decisions fail when we conflate lack of adoption with a lack of need.
A better approach is to ask whether the job we are solving is worth solving, and if so, do we have tools that can do the job better than the product we are reviewing. Without that, we see the same products getting rebuilt with each change of guard.
5. Sharpen our saw
To accelerate our learning loops, we need better tools to detect issues, measure them, and improve our ability to fix them.
Some of the ways we sharpen our saw, is by creating tools that help us raise the bar:
- Identify. Accelerate how fast we can identify issues through alerts when things fail in production. Heuristic for triaging and evaluating dogfooding feedback.
- Diagnose. Understand why something is broken by adding instrumentation that enables root cause analysis. Leverage in product surveys to get direct signal.
- Resolve. Raise the bar with every fix by building scalable design patterns and better testing coverage.
6. (Bonus) Create better incentives
The behavior we reward is the culture we create. Each of the above is a tactical step towards improving quality. However, the spark for making it happen is the culture that rewards people for acknowledging and addressing issues through initiative such as:
- Reward people for work on improving gritty areas such as reliability and performance. These are easy to criticize but hard to move and even harder to motivate people to work on.
- Create objective measures for tracking quality. Include both internal scores such as crash rate, and public lists in forums such as UserVoice. These are easy to reference and goal against.
- Celebrate unshipping low value products. Removing low quality products should be cheered on and acknowledged just as much as adding features to avoid creating an increasingly complicated product.
If you are in the trenches, working on improving product quality, I would love to hear what else has worked for you, and always happy to share anything else from our efforts.