mlsanityData to 14 Oct 2026

Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant

arXiv 2605.06240 ↗First posted 7 May 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Forward-Forward (FF) training lets each layer learn from a local goodness criterion. In cumulative-goodness variants, later layers can inherit a task that earlier layers have partly separated. We formalize this as layer free-riding: under the softplus FF criterion, the class-discrimination gradient reaching block d decays exponentially with the positive margin accumulated by earlier blocks (Theorem 3.1); a squared-hinge barrier reproduces the pathology. History-free and hardness-gated repairs raise deeper-layer separation by up to 5× on CIFAR-100 and 45× in an 8-block CIFAR-10 model, yet these and a depth-scaled auxiliary term move accuracy by under one percentage point among non-degenerate variants; on Tiny ImageNet, a harder cross-dataset check of the selected configuration, layer health likewise does not track accuracy. A theory-derived attenuation-compensated objective raises deepest-block separation 5.9× at a consistent cost in the FF classifier's own accuracy that a probe trained on the frozen features recovers; in a smaller CIFAR-10 control, most of its separation gain comes from dropping the depth-scaled auxiliary term it replaces. Accuracy invariance is a property of the deployed sum-based scoring rule under redistribution (Proposition 3.2) and, empirically, of the structured trained readouts we test, not of what the network learns: retrained variants disagree on 13-26% of test predictions. Architecture and augmentation matter far more than the non-degenerate training rules studied. Cumulative free-riding is therefore a real and repairable pathology, but for the softplus and squared-hinge objectives, architectures, and datasets we study, it is not the dominant factor limiting accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 40 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

13 of 40 references.

References · 13

  1. 2020GLU Variants Improve Transformercited by 1,804
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  3. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 266
  4. 2023Convolutional Channel-wise Competitive Learning for the Forward-Forward Algorithmcited by 25
  5. 2023Layer Collaboration in the Forward-Forward Algorithmcited by 21
  6. 2023The Trifecta: Three simple techniques for training deeper Forward-Forward networkscited by 12
  7. 2024Distance-Forward Learning: Enhancing the Forward-Forward Algorithm Towards High-Performance On-Chip Learningcited by 8
  8. 2023Extending the Forward Forward Algorithmcited by 6
  9. 2025Contrastive Forward-Forward: A Training Algorithm of Vision Transformercited by 6
  10. 2025Adaptive Spatial Goodness Encoding: Advancing and Scaling Forward-Forward Learning Without Backpropagationcited by 3
  11. 2026Hyperspherical Forward-Forward with Prototypical Representationscited by 3
  12. 2025Scalable Forward-Forward Algorithmcited by 2
1 more
  1. 2024Cyclic Neural Networkcited by 1

Built on by

Not cited yet.