Gauge-Fixing the Forward-Forward Objective: A Whitened Goodness Derived from a Likelihood-Ratio Account
Not on the map. Why?
Abstract (arXiv)
The Forward-Forward algorithm trains each layer locally, so that a scalar goodness - the sum of squared activations - is high on real inputs and low on contrastive ones. Under an explicit generative model this goodness is the sufficient statistic of a likelihood-ratio test, and the pairwise form of the objective admits a gauge: a layer can lower its loss by inflating the scale of its weights rather than by separating the two populations. The analysis prescribes the repair - a whitened, scale-invariant goodness trained online within each layer - which we evaluate as a training procedure. Across three corpora, three depths and a fourfold range of layer width (13 seeds per cell), it raises linear-probe accuracy over the standard pairwise objective in every measured cell - by 4 to 7 points on eight of nine corpus-depth combinations - and closes 16-61% of the gap to end-to-end backpropagation. A control isolates the mechanism: Hinton's fixed-threshold loss also bounds the runaway, to a factor of 1.4 against 133, yet tracks the unmodified baseline - invariance to the gauge, not a bound on it, is what pays. Against the strongest published alternative - a sparse, top-k goodness - the derived objective is statistically indistinguishable on two corpora of three, yet only it removes the runaway: sparsity and gauge-invariance are independent axes, and the published variant recovers accuracy while leaving the pathology in place. We state the boundaries we measured, and every prediction was recorded before its experiment with the refutations reported.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 28 references that are in the map. First posted on arXiv 14 Jul 2026.
Builds on
13 of 28 references.
References · 13
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 266
- 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
- 2019Training Neural Networks with Local Error Signalscited by 136
- 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 115
- 2019Putting An End to End-to-End: Gradient-Isolated Learning of Representationscited by 86
- 2023The Predictive Forward-Forward Algorithmcited by 35
- 2023Layer Collaboration in the Forward-Forward Algorithmcited by 21
- 2023The Cascaded Forward Algorithm for Neural Network Trainingcited by 17
- 2025In Search of Goodness: Large Scale Benchmarking of Goodness Functions for the Forward-Forward Algorithmcited by 2
- 2026Covariance-Aware Goodness for Scalable Forward-Forward Learningcited by 2
Built on by
Not cited yet.