Small a/b test puzzle that broke my brain

Our take

In the world of data science, even small A/B tests can unveil surprising insights. While testing two homepage banners, I encountered a puzzling scenario: Banner A appeared to win overall, yet when segmented by device, Banner B outperformed both on desktop and mobile. This discrepancy highlighted an important lesson about data integrity—Banner A's success stemmed from a more favorable audience mix rather than superior design. This experience, rooted in Simpson's Paradox, underscores the importance of thorough data analysis.

I recently build a platform, that aims to help everyone to practice data science cases, to get hands on experience. I've been working as a DS for years. I mainly use databricks or Hex notebook, with AI assistant. So this platform can let you practice with the same tool. This is one of the best case I've built, and I want to share it with all of you---

Imagine you're testing two homepage banners. banner A vs banner B. Two weeks of traffic, lots of data. Banner A wins by a comfortable margin - cool, ship A, done.
Then for some reason you decide to split it by device before pushing the button.
Desktop: B wins
Mobile: B wins

So, banner B is better for desktop users. And banner B is better for mobile users. But added up, banner A wins overall? How the answer is the test wasn't fair. For whatever reason (caching, ad targeting, just bad luck), banner A got shown to a lot more desktop traffic than banner B did. And desktop users convert way better than mobile on almost every site. So it turns out A wasn't a better banner, it was a banner that got tested on an easier audience. Fix the traffic mix and B is the right call.

This thing has a name (Simpson's Paradox if you wanna google it) but you don't need the name to spot it. you just need to remember to slice your data before you trust the headline. If you are interested, you can practice the same case at https://www.litmetrics.ai/

submitted by /u/Alarming-Wish207
[link] [comments]

Tagged with

#generative AI for data analysis#Excel alternatives for data analysis#natural language processing for spreadsheets#big data management in spreadsheets#conversational data analysis#real-time data collaboration#financial modeling with spreadsheets#intelligent data visualization#data visualization tools#enterprise data management#big data performance#data analysis tools#data cleaning solutions#rows.com#google sheets#A/B test#data science#traffic#Simpson's Paradox#banner A