Choosing the right causal model is a skill that separates senior data scientists from the rest, yet it's rarely taught with the rigor we apply to machine learning. This simulation makes that point painfully clear: when Claude Code was given free rein over a dataset with a known 10% treatment effect, it anchored on difference-in-differences and reported a 24% increase, more than double the truth. The model even flagged that pre-treatment trends weren't parallel, but it never questioned whether a different approach might fit better. That's not a failure of the AI; it's a mirror of common human practice. We default to familiar tools and then trust their output because they look reasonable. The practical fix is simple and often overlooked: run a placebo test before you trust any model, and let that guide your selection. This workflow echoes the discipline we've discussed in How to Spot the 4 Case Study Types in Data Science Interviews, where the right framing often matters more than the right formula.
The A/A test, or placebo test, is the unsung hero here. You take only pre-treatment data, pretend the treatment happened earlier, and randomly assign five regions as "treated" when nothing actually occurred. The true effect is zero, so any estimate the model produces is pure error. Repeating this 500 times gives you a spread of that error. For difference-in-differences, 90% of placebo estimates fell between -14% and +10%, with a root mean square error of 7.6 percentage points. Synthetic control on the same draws tightened that range to -7% to +9%, dropping the RMSE to 4.9 points. Even the outcome variable matters: switching to bookings per 1,000 people reduced the error further to 4.4 points. When applied to the real treatment period, synthetic control estimated 7% on raw bookings and 9.6% per capita, both far closer to the true 10% than the 24% from difference-in-differences. This is exactly the kind of decision-making that How AI agents are reshaping the data scientist's toolset highlights: the tools are getting smarter, but the responsibility for validation still rests with us.
What carries over to real work is the workflow, not the winner. One simulated dataset doesn't prove synthetic control always beats difference-in-differences, and the gap between the two synthetic control versions is too small to generalize. But the placebo test is universally applicable. In practice, you never see the ground truth, so you need a way to measure estimation error before committing to a model. That means building a routine where you test your candidate models on fake treatments, compare their error distributions, and then choose the one that performs best in the absence of a real signal. It's a habit that costs little time and can save you from reporting an effect that's more than double the actual impact. As we've noted in Make Smarter AI Agents That Know When to Escalate and When to Act, knowing when to pause and verify is often the difference between a useful analysis and a misleading one. The concrete takeaway: before you trust any causal estimate, run the placebo test on every model you're considering, and let the error spread make the choice for you.