Visualizing Loss Landscapes of Neural Networks [P]

Hey r/MachineLearning,

Visualizing the loss landscape of a neural network is notoriously tricky since we can't naturally comprehend million-dimensional spaces. We often rely on basic 2D contour analogies, which don't always capture the true geometry of the space or the sharpness of local minima.

I built an interactive browser experiment https://www.hackerstreak.com/articles/visualize-loss-landscape/ to help build better intuitions for this. It maps how different optimizers navigate these spaces and lets you actually visualize the terrain.

To generate the 3D surface plots, I used the methodology from Li et al. (NeurIPS 2018). This is entirely a client-side web tool. You can adjust architectures (ranging from simple 1-layer MLPs up to ResNet-8 and LeNet-5), swap between synthetic or real image datasets, and render the resulting landscape.

A known limitation of these dimensionality reductions is that 2D/3D projections can sometimes create geometric surfaces that don't exist in the true high-dimensional space. I'd love to hear from anyone who studies optimization theory and how much stock do you actually put into these visual analysis when analysing model generalization or debugging.