mlsanityData to 7 Oct 2026

Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation

arXiv 2406.05222 ↗First posted 7 Jun 2024cs.LG cs.NEcited by 4

Not on the map. Why?

Abstract (arXiv)

Relieving the reliance of neural network training on a global back-propagation (BP) has emerged as a notable research topic due to the biological implausibility and huge memory consumption caused by BP. Among the existing solutions, local learning optimizes gradient-isolated modules of a neural network with local errors and has been proved to be effective even on large-scale datasets. However, the reconciliation among local errors has never been investigated. In this paper, we first theoretically study non-greedy layer-wise training and show that the convergence cannot be assured when the local gradient in a module w.r.t. its input is not reconciled with the local gradient in the previous module w.r.t. its output. Inspired by the theoretical result, we further propose a local training strategy that successively regularizes the gradient reconciliation between neighboring modules without breaking gradient isolation or introducing any learnable parameters. Our method can be integrated into both local-BP and BP-free settings. In experiments, we achieve significant performance improvements compared to previous methods. Particularly, our method for CNN and Transformer architectures on ImageNet is able to attain a competitive performance with global BP, saving more than 40% memory consumption.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

42 of 56 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  6. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  8. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  9. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  10. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  11. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  12. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 215
30 more
  1. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 135
  2. 2019Training Neural Networks with Local Error Signalscited by 135
  3. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 113
  4. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 112
  5. 2018Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architecturescited by 100
  6. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  7. 2019Deep Learning without Weight Transportcited by 87
  8. 2019Putting An End to End-to-End: Gradient-Isolated Learning of Representationscited by 85
  9. 2019Decoupled Greedy Learning of CNNscited by 76
  10. 2022Gradients without Backpropagationcited by 73
  11. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  12. 2015Towards Biologically Plausible Deep Learningcited by 66
  13. 2022Scaling Forward Gradient With Local Lossescited by 65
  14. 2022Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Passcited by 63
  15. 2016Training Neural Networks Without Gradients: A Scalable ADMM Approachcited by 61
  16. 2022Hebbian Deep Learning Without Feedbackcited by 61
  17. 2015How Important is Weight Symmetry in Backpropagation?cited by 57
  18. 2020LoCo: Local Contrastive Representation Learningcited by 55
  19. 2020L2-GCN: Layer-Wise and Learned Efficient Training of Graph Convolutional Networkscited by 48
  20. 2020Local plasticity rules can learn deep representations using self-supervised contrastive predictionscited by 47
  21. 2023Understanding Imbalanced Semantic Segmentation Through Neural Collapsecited by 46
  22. 2017Deep supervised learning using local errorscited by 41
  23. 2018Biologically-plausible learning algorithms can scale to large datasetscited by 37
  24. 2022A Law of Data Separation in Deep Learningcited by 36
  25. 2023Blockwise Self-Supervised Learning at Scalecited by 22
  26. 2022Problem-Dependent Power of Quantum Neural Networks on Multi-Class Classificationcited by 20
  27. 2021Credit Assignment Through Broadcasting a Global Error Vectorcited by 19
  28. 2023Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variantscited by 14
  29. 2022Towards Theoretically Inspired Neural Initialization Optimizationcited by 11
  30. 2019Effects of Depth, Width, and Initialization: A Convergence Analysis of Layer-wise Training for Deep Linear Neural Networkscited by 4

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
  2. 2025Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergencecited by 4
  3. 2025Stochastic Layer-wise Learning: Scalable and Efficient Alternative to Backpropagationcited by 1
  4. 2025DeInfoReg: A Decoupled Learning Framework for Better Training Throughputcited by 1