# Momentum Auxiliary Network for Supervised Local Learning

arXiv: https://arxiv.org/abs/2407.05623 · HTML: https://mlsanity.com/paper/2407.05623/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 8 Jul 2024.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [Difference Target Propagation](https://mlsanity.com/paper/1412.7525/) (2014)
- [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016)
- [Training Neural Networks with Local Error Signals](https://mlsanity.com/paper/1901.06656/) (2019)
- [Greedy Layerwise Learning Can Scale to ImageNet](https://mlsanity.com/paper/1812.11446/) (2018)
- [Deep Learning without Weight Transport](https://mlsanity.com/paper/1904.05391/) (2019)
- [Putting An End to End-to-End: Gradient-Isolated Learning of Representations](https://mlsanity.com/paper/1905.11786/) (2019)
- [How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation](https://mlsanity.com/paper/1407.7906/) (2014)
- [Decoupled Greedy Learning of CNNs](https://mlsanity.com/paper/1901.08164/) (2019)
- [Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction](https://mlsanity.com/paper/2103.04174/) (2021)
- [Revisiting Locally Supervised Learning: an Alternative to End-to-end Training](https://mlsanity.com/paper/2101.10832/) (2021)
- [Towards Biologically Plausible Deep Learning](https://mlsanity.com/paper/1502.04156/) (2015)
- [Scaling Forward Gradient With Local Losses](https://mlsanity.com/paper/2210.03310/) (2022)
- [Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Pass](https://mlsanity.com/paper/2201.11665/) (2022)
- [LoCo: Local Contrastive Representation Learning](https://mlsanity.com/paper/2008.01342/) (2020)
- [Local plasticity rules can learn deep representations using self-supervised contrastive predictions](https://mlsanity.com/paper/2010.08262/) (2020)
- [Deep supervised learning using local errors](https://mlsanity.com/paper/1711.06756/) (2017)
- [Parallel Training of Deep Networks with Local Updates](https://mlsanity.com/paper/2012.03837/) (2020)
- [Credit Assignment Through Broadcasting a Global Error Vector](https://mlsanity.com/paper/2106.04089/) (2021)
- [Interlocking Backpropagation: Improving depthwise model-parallelism](https://mlsanity.com/paper/2010.04116/) (2020)

## Built on by (most cited first, 10 of 10)

- [Faster Multi-GPU Training with PPLL: A Pipeline Parallelism Framework Leveraging Local Learning](https://mlsanity.com/paper/2411.12780/) (2024)
- [Rethinking Local Learning: A Cheaper and Faster Recipe for LLM Post-Training](https://mlsanity.com/paper/2605.04913/) (2026)
- [MAN++: Scaling Momentum Auxiliary Network for Supervised Local Learning in Vision Tasks](https://mlsanity.com/paper/2507.16279/) (2025)
- [NeuCLIP: Efficient Large-Scale CLIP Training with Neural Normalizer Optimization](https://mlsanity.com/paper/2511.08417/) (2025)
- [LAKD-Activation Mapping Distillation Based on Local Learning](https://mlsanity.com/paper/2408.11478/) (2024)
- [Replacement Learning: Training Vision Tasks with Fewer Learnable Parameters](https://mlsanity.com/paper/2410.01239/) (2024)
- [Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards](https://mlsanity.com/paper/2605.03862/) (2026)
- [Depth-Wise Representation Development Under Blockwise Self-Supervised Learning for Video Vision Transformers](https://mlsanity.com/paper/2601.09040/) (2026)
- [Replacement Learning: Training Neural Networks with Fewer Parameters](https://mlsanity.com/paper/2605.19533/) (2026)
- [Greedy Local Learning for Language Model Pretraining: Gaps and Objective Design](https://mlsanity.com/paper/2610.04867/) (2026)

## Abstract (arXiv)

Deep neural networks conventionally employ end-to-end backpropagation for their training process, which lacks biological credibility and triggers a locking dilemma during network parameter updates, leading to significant GPU memory use. Supervised local learning, which segments the network into multiple local blocks updated by independent auxiliary networks. However, these methods cannot replace end-to-end training due to lower accuracy, as gradients only propagate within their local block, creating a lack of information exchange between blocks. To address this issue and establish information transfer across blocks, we propose a Momentum Auxiliary Network (MAN) that establishes a dynamic interaction mechanism. The MAN leverages an exponential moving average (EMA) of the parameters from adjacent local blocks to enhance information flow. This auxiliary network, updated through EMA, helps bridge the informational gap between blocks. Nevertheless, we observe that directly applying EMA parameters has certain limitations due to feature discrepancies among local blocks. To overcome this, we introduce learnable biases, further boosting performance. We have validated our method on four image classification datasets (CIFAR-10, STL-10, SVHN, ImageNet), attaining superior performance and substantial memory savings. Notably, our method can reduce GPU memory usage by more than 45% on the ImageNet dataset compared to end-to-end training, while achieving higher performance. The Momentum Auxiliary Network thus offers a new perspective for supervised local learning. Our code is available at: https://github.com/JunhaoSu0/MAN.