Product Manager Math Part 2
A few days ago I wrote about an insidious math error made frequently by product managers. Today I’d like to talk about setting success…
A few days ago I wrote about an insidious math error made frequently by product managers. Today I’d like to talk about setting success criteria for projects and experiments.
For example, how do we decide what impact on conversion an experiment ought to have in order for us to deem it successful?
I’ve read a bunch about the importance of having success criteria, but less about how to do it. There’s no simple answer, and nobody wants to write down the obvious. But one thing I have learned is that nothing is obvious when it comes to Product Management.
There is a data science aspect to this. If you are fortunate enough to have data scientists to work with on your team, they should be involved in your experiment design and determining your success criteria, not just the analysis of your outcomes.
One technique I have learned is the importance of A-A testing, which is when you use your testing apparatus to run an experiment in which two user cohorts get the same experience. This tests your test for sources of bias or other errors that could influence the outcome of your experiment.
Even in well-designed and thoughtfully controlled experiments, randomness, errors, and the biases you cannot control for will contaminate your results. Different people set different thresholds for this, and I’m not sure it’s scientific but I have adopted the policy that any result less than 5% is just the universe tampering with my soul. For me, success starts at 6%.
That is the bare minimum impact that I’m hoping to have from a small, inexpensive experiment created by a small team. For bigger ventures, we need more math.
One tactic is to think about ROI. If we design an experiment that requires a significant investment, we should require it to have a higher return. Start by estimating the cost of the resources we’re investing, and subtracting the return we are hoping for (e.g. the lifetime value of new subscribers.) Your finance department can help you here. Perhaps a successful project should pay for itself within a year, or maybe three.
Another consideration is opportunity cost. Opportunity cost is an essential concept in economics but perhaps not discussed frequently enough in Product Management: It’s the tradeoff between choices.
If you choose to build Feature A, you must forego the benefits of Feature B. If Feature B has the same ROI as A, then no opportunity cost. But if Feature B had higher ROI than A, you incur the opportunity cost of that decision.
If you have a team dedicated to onboarding experiments, then assigning them another may not have much opportunity cost. On the other hand, if you want to implement 2FA and that’s going to occupy an core infrastructure team for four months — preventing them from doing anything for any other teams… Your ROI had better cover the opportunity cost.
This is a chapter-length topic crammed into a short post. I hope I’ve shed at least a little light on it. Good luck with your planning and your projects!