When I first became a product analyst I assumed product experiments were similar to ones I evaluated as a marketing data analyst – but I was wrong. When a marketing A/B test wasn’t statistically significant, we moved on to the test but after working on product A/B tests, I realized even ones with no statistical significance provided valuable insights. Today I want to review examples of product A/B tests that didn’t perform as expected and lessons learned.
Unexpected Results
Hypothesis
We launched a new meal plan feature with the hypothesis that it would increase the trial to paid ( TTP ) conversion rate. A higher TTP rate meant more users would convert to paying subscribers translating to increased revenue. The test was run with two groups divided 50/50 with the test group having access to the new feature and control group having no access.
Results
The new feature showed no improvement in the overall TTP conversion rate. This was counterintuitive because the plan feature was the most frequently requested feature from users for years and I expected to see a higher TTP rate. I decided the test group warranted further analysis to understand why the TTP rates were similar before presenting the results to the product manager.
I divided the test __ group into different segments to see if there were noticeable differences in TTP rates. Group A were users that knew about the plan feature after starting a trial while group B were users that didn’t know about it. I further divided group A into users that started a plan and ones that didn’t.
After dividing the test group into the segments shown above I could see noticeable differences in TTP rates. Group B had the same TTP rate as control of 30% because these users weren’t aware the plan feature existed similar to control. Group A users that never started a plan had a 35% TTP rate compared to users that started a plan with a TTP rate of 25%. Why the big difference between these two Group A segments?
Since the plan feature was 28 days long, users forgot about it and didn’t experience the impact they would’ve if they have completed the plan. This was supported when we looked at the low plan completion rate. Due to this poor user experience, it made sense users were less likely to pay after their trial ended. This was a valuable insight for the product manager to prioritize plan notifications to remind users to complete their weekly tasks which would translate to higher completion rates and increased TTP rates.
Now, why would Group A users that knew about plans but didn’t try one have a higher TTP rate? We hypothesized this indicated users liked the plans feature and although they didn’t try it the perceived value convinced them to pay after their trial ended and they could try plans later. Remember this was a highly requested feature from users.
Takeaways
- Test results are not always what they seem. If results are unexpected, try segmenting users by different actions to identify any differences in conversion rates. This can surface actionable insights for the product manager as in this example it was to add plan notifications that would increase plan completion rates thereby increasing TTP rates.
- Check if there are external factors that may affect test results. For the A/B test, we coordinated with product marketing to hold off notifying users about the plan feature until we were done testing. This may not be the case for tests you support and you should be mindful of possible impacts.
Unintentional Side Effects
Hypothesis
Past analysis showed users that who logged a food after signing up for the app were more likely to return compared to users that didn’t. A change was added to the signup process to encourage users to log a food after the signup process was completed to improve user engagement.
Results
The test ran for 4 months but didn’t show any improvement in user engagement and it was turned off. Around the same time, we started seeing a drop in trial starts. I supported subscriber analytics and wasn’t aware of this test. My team couldn’t figure out why trials were decreasing until the product manager hypothesized it might have been caused by the engagement test.
Once we turned on the test again, trials starts went back up to prior levels. Since the test had run for 4 months we had attributed trial increases to the growth of the business and not to a test that wasn’t supposed to affect trials at all. It turned out the product manager had switched the upgrade screen to another part of the signup process for the test variant and this caused the trial starts to increase. The screen change wasn’t done on purpose but this was a valuable insight we learned about where to position the upgrade message in order to increase trial starts.
Takeaways
- Tests may have unintended impacts on other KPIs the product manager isn’t using as a measure of success. Consider all KPIs a test may impact to make sure there are no negative impacts to any of them. In my example, there was no change in engagement but a significant positive impact on trial starts.
- Confirm control and test variants have the same elements other than the changes being tested. We were lucky with our test that the unintended side effects caused trial starts to increase but it could’ve easily gone the other way and caused a decrease instead.
- Be aware of planned experiments even if you’re not supporting them. Talk to product managers about their product roadmap and find out the initiatives they plan to work on. If I had talked to product managers outside of subscriptions I might have learned about the engagement test earlier.
Final Thoughts
While it was challenging at first, I learned a great deal about product A/B tests as a product analyst. Whether you’re new to product or thinking about becoming a product analyst, I hope this gives you a head start on your A/B testing experience. Thanks for reading!
You might also like…
What Data Analysts Should Know About Product Analytics
