Retraining by the calendar? Real fraud data says rethink your triggers.

In "Why MLOps Retraining Schedules Fail — Models Don’t Forget, They Get Shocked," we delve into the shortcomings of traditional calendar-based retraining for machine learning models.

3 min readTowards Data Science
Retraining by the calendar? Real fraud data says rethink your triggers.

The calendar is a convenient scapegoat for model decay, but the data tells a different story. When the Ebbinghaus forgetting curve was fitted to 555,000 real fraud transactions, the result was an R² of −0.31. That is not just a weak correlation; it is worse than a flat line. A model that never retrained would have performed more predictably than one following a schedule based on time. This is the clearest evidence yet that the problem is not forgetting. It is shock.

What does that mean for you, the person responsible for keeping a fraud model alive in production? It means your weekly or monthly retraining ritual is likely addressing a symptom that does not exist. The model is not drifting away from its training data because of some natural cognitive decay. It is being shoved off a cliff by sudden, structural changes in the data environment. A new merchant category, a coordinated attack, a shift in how payments clear: these are shocks. They do not arrive on a schedule, and no amount of calendar-based retraining will catch them in time. You are training for a war that ended, while the actual battle is happening elsewhere.

The practical takeaway is not to abandon retraining entirely. It is to replace the trigger. Instead of asking "when did we last train?", ask "has the world changed enough that our model is now blind?" Shock detection is not about plotting loss curves over time and waiting for a dip. It is about monitoring the input distribution for sudden, meaningful deviations and retraining only when those occur. That approach is cheaper, faster, and directly tied to the conditions that actually degrade performance. The R² of −0.31 is not an abstract statistic; it is a warning that your current schedule is not suboptimal, it is actively counterproductive.

The next time you are tempted to defend a retraining cadence because "that is how we have always done it," remember the flat line you are already outperforming. The calendar is a crutch, and the data has knocked it away. Build your monitoring around shocks, not dates, and your model will finally be responding to the world as it is, not as a schedule pretends it should be.

From Towards Data Science

We fitted the Ebbinghaus forgetting curve to 555,000 real fraud transactions and got R² = −0.31 — worse than a flat line. This result explains why calendar-based retraining fails in production and introduces a practical shock-detection approach that works in real systems.

The post Why MLOps Retraining Schedules Fail — Models Don’t Forget, They Get Shocked appeared first on Towards Data Science.

Read the original at Towards Data Science