Functional Gradient Descent

Solving functional gradient descent with adaptive representations

Functional gradient descent often beats neural networks in practice, yet it rarely gets implemented accurately. The core issue is that functional gradients are infinite-dimensional, and naive approximations converge to the wrong solution. This new work, accepted at NeurIPS, formalizes a class of adaptive representations that provably guarantee convergence to the global minimizer. The resulting algorithms outperform neural nets by an order of magnitude across multiple settings. It's early days, but this line of research carries genuine potential.