The problem this engineer describes is real, and it is the kind of tension that defines serious machine learning work: you need hyperparameter optimization (HPO) to achieve high accuracy, but the full training run is so long that you cannot afford to do it for every trial. Reducing epochs and adding pruning are sensible shortcuts, but they introduce a new problem, the parameters that look best on a short run may not be the same ones that perform best when the model is allowed to converge fully over a full day. The fear of parameter drift is not irrational. The solution is not to abandon HPO, but to rethink how you define the relationship between the abbreviated search and the final training.
Here is what we think: if you are reducing epochs and using median pruning, you are optimizing for early performance, not final performance. A learning rate scheduler tuned for a 1- or 2-hour trial is unlikely to match the behavior of a scheduler designed for a 24-hour run. The early training dynamics are different, the model is still in a high-gradient regime, and the loss surface is steep. What looks like a good learning rate schedule in the first few epochs may be too aggressive or too conservative for the later stages of full training. The idea of restarting the learning rate scheduler after the model stops learning is worth testing, but be careful: a simple restart without adjusting the cycle length can destabilize convergence. Instead, consider a cosine annealing schedule with warm restarts, which is designed for exactly this scenario, it allows the model to escape local plateaus while retaining past progress.
Median pruning favors trials that converge quickly. That is its strength and its limitation. It does not punish slower convergence outright, but it does stop trials that fall below the median performance at the current step. If a trial would have surpassed the others later, it never gets the chance. This is a real risk when the best parameters only show their advantage in the second half of training. To mitigate this, you have a few options. Increase the minimum number of epochs before pruning begins, so every trial has time to escape the noise of early training. Alternatively, use a percentile pruner with a wider acceptance window, which keeps more trials alive longer without eliminating the speed benefit entirely. You can also run a validation step after full training that compares a subset of pruned and unpruned trials from the same HPO run to measure the false-negative rate in your pruning strategy.
The practical takeaway is this: build a two-stage pipeline. First, run a short HPO with a lightweight scheduler and a generous pruning policy to narrow down the hyperparameter space. Then, take the top 5-10 candidates and run them with the full epoch count and the real scheduler. You lose some of the speed advantage, but you gain confidence that the parameters you select are actually the ones that work best at full scale. That tradeoff is worth it when accuracy is the priority and retraining happens twice a month across five models.