Experimenting

I had a conversation with a product manager the other day that I can’t stop thinking about. It was about onboarding and experimentation.

I had a conversation with a product manager the other day that I can’t stop thinking about. It was about onboarding and experimentation.

This person (who is awesome) was working on an onboarding user flow. They’d done some great experiments that seemed to support the hypothesis that shortening the flow would increase the completion rate and ultimately lift conversion.

They’re getting good results this way, and the PM is planning more experiments to further optimize the experience. I suggested that one of those experiments might be to be eliminate this experience altogether, to confirm that it is making *any* positive contribution to our metrics.

They responded that they didn’t think this experiment would be possible. I’m oversimplifying the conversation, but part of the flow is to collect information used to set defaults and so on, and they didn’t think this step could easily be skipped.

If we believe that pre-configuring the product is important, but we can’t easily confirm that belief with our metrics, something is wrong. Either our priorities are inverted, or we’re measuring the wrong things.

I saw an interview with a bajillionare founder who said something like “Engineers have a tendency to optimize what they should eliminate.” I don’t really like that guy, but I do love this sentiment.

Obviously onboarding is important. But we should remember that the point of A/B testing is to test the performance of an alternative versus a control group, and that alternative does not have to be additive to the status quo.

In a scientific experiment, we might test how adding a supplement at mealtimes impacts iron levels in the blood. We do that by separating a group into control and experimental groups, and giving the first group nothing or a placebo, giving the second group the supplement, and measuring and comparing the outcomes.

But we could also test how fasting impacts iron level. Same control, but nothing is given to the experimental group. Instead, we take away their entire meal! It’s poor hospitality but it’s frugal, and it might work better than the supplement. It’s a valid and meaningful experiment.

A product that’s a few years old has probably had things added to it many, many times. The FDA does not require that our experiments start with the assumption that something must be added to the status quo. We can also take things away.

I think it’s a good idea to try to run experiments that ask “What if we just got rid of this completely?”

What do you think?