sum of outputs

Streamlining softmax by reducing inputs without sacrificing accuracy

Reducing softmax's inputs from N to N-1 is a neat theoretical insight, those redundant parameters are doing nothing. If the logits sum to zero, the last one is already determined. The idea could trim the final layer and maybe nudge convergence. But the benefit feels negligible in practice, as the author suspects. Why not do it anyway? Because elegance doesn't always translate to speed. For a deeper look at streamlining neural architectures, our article on "Functional Gradient Descent with Adaptive Representations" explores similar efficiency gains.