Are there any theoretically-guided practices left in machine learning nowadays? [D]

The recent Reddit post questioning the relevance of theoretically-guided practices in modern machine learning strikes a resonant chord within our community. It highlights a shift away from a rigorous, mathematically-grounded approach toward a more empirical, experimentation-driven methodology. The author’s observation about the evolution of established “theories” – from overfitting prevention to the sanctity of the test set – into almost unquestioned dogma, only to be quietly abandoned by their originators, is particularly insightful. This echoes a broader trend we've seen across the field, where the rapid pace of innovation often outstrips the ability to develop robust theoretical underpinnings. This isn’t necessarily a negative; the pragmatism has undeniably fueled impressive advancements. However, it does raise questions about the long-term sustainability of a field increasingly reliant on trial and error, as explored in a related discussion about the impact of honest limitations sections on paper reviews How much does adding an honest limitations section hurt the paper?. The increasing use of agentic reviewers, as detailed in For the people who got reviews back from neurips, cvpr, eccv, etc and also tested their paper through an agentic reviewer like the stanford one, how different were the reviews?, further underscores this shift; reviewers are increasingly evaluating models based on their observed performance rather than scrutinizing the theoretical basis of their design.

The post’s central question – are there *any* remaining theoretically-guided practices – is a challenging one. While the days of blindly adhering to rules like “never look at the test set” are largely over, theoretical considerations haven't vanished entirely. Optimizer selection, for instance, still often reflects some understanding of the underlying landscape being optimized, even if that understanding is less formalized than it once was. The rise of techniques like contrastive learning, with its roots in information theory, demonstrates that theoretical grounding can still inspire impactful innovations. What’s changed is the *degree* to which theory dictates practice. The emphasis has moved from deductive reasoning – starting with a theoretical principle and applying it – to inductive reasoning – observing patterns in empirical results and formulating hypotheses. The development of tools like PyTorch’s ‘torch-preflight’ A linter for PyTorch 'torch-preflight' highlights a growing need for more robust debugging and validation techniques, hinting at a desire to reintroduce some rigor into the development process.

This empirical turn isn’t inherently problematic. The sheer scale of modern datasets and computational power allows for experimentation that was simply impossible in earlier eras. The ability to rapidly iterate and test different architectures and training regimes has undoubtedly accelerated progress. However, a complete reliance on empirical results carries risks. It can lead to overfitting to specific datasets, a lack of generalizability, and a difficulty in diagnosing and correcting errors. Without a solid theoretical foundation, it becomes harder to understand *why* a particular model works, making it difficult to adapt or improve it in the future. Furthermore, the lack of theoretical understanding can make it challenging to transfer learnings across different domains or tasks. The current landscape is characterized by a vast number of models achieving impressive results on benchmarks, but with limited insight into their underlying mechanisms.

Ultimately, the future of machine learning likely lies in a synthesis of theoretical rigor and empirical exploration. We need to move beyond simply optimizing for performance on specific benchmarks and cultivate a deeper understanding of the principles that govern model behavior. This means not discarding theory entirely, but rather embracing a more nuanced approach where theoretical insights inform empirical investigations, and empirical results, in turn, refine and challenge existing theories. The question isn't whether theory is dead, but how we can best integrate it into a rapidly evolving and increasingly complex field. What new theoretical frameworks will emerge to explain the successes – and failures – of the large language models and other powerful AI systems that are reshaping our world?