Population Based Training is a rescue tool, not a speed-up. That is the blunt, useful conclusion from a careful set of experiments on grokking, and it reframes how we should think about adaptive hyperparameter methods. The researcher ran PBT over learning rate and weight decay on modular addition, comparing it against a plain grid with equal compute. The grid's best fixed setting hit 95% validation accuracy in about 317 steps. PBT took roughly 2,667 steps and was slower in every single seed. When the starting range was a dead zone, where no fixed hyperparameter setting groks at all, PBT did eventually escape, but only with aggressive mutation every 100 steps, and it took around 6,850 steps to do it. A wide random search from the same starting conditions found the answer at a median of 300 steps.
The deeper insight is that PBT's success in the dead zone comes from the schedule it discovers, not from the population dynamics. When the winning learning rate and weight decay trajectory was replayed on fresh initializations without any PBT machinery, 55 out of 60 runs reached 95% accuracy. Holding those knobs constant at their geometric mean produced zero successes out of 60. That is a clean, direct result: the path matters, not the copying of weights. The val loss initially rewards low learning rate times weight decay, the wrong direction for grokking, and then flips sign later. PBT is essentially navigating a landscape that changes shape mid-run, which is why it can find an escape route that static search misses. But it pays a heavy price in compute, and it can also collapse into the floor-ceiling corner and never leave.
This should change how practitioners allocate effort. If you have a reasonable hyperparameter range, a grid or random search will beat PBT on time-to-accuracy. If you are stuck in a range where nothing works, PBT is a legitimate escape hatch, not because it is faster but because it can discover a non-stationary schedule that a fixed setting cannot express. That aligns with a pattern we have seen elsewhere in this publication: Spline-based regression outpaces MLPs in fitting 4D Mandelbulb data shows how a specialized inductive bias can outperform a general-purpose model on a specific task, and Explore how EpochGo brings local-first AutoML to your data with privacy and control argues for tools that respect your constraints rather than forcing a one-size-fits-all pipeline. Here, the constraint is the hyperparameter range itself.
The caveats are honest and important: one task, one model, one split, and p-values mostly above 0.05. The replayed schedules are post-hoc winners, and the author has not yet tested a range with no grokking region at all, which is exactly where PBT should theoretically beat random search. That is the open question worth watching. If you are debugging a model that refuses to train, do not assume adaptive methods will save you compute. They might save your run, but only if you are willing to let them wander. The next experiment should test a genuinely barren range, and we hope someone runs it.