# Module-wise Training of Neural Networks via the Minimizing Movement Scheme

arXiv: https://arxiv.org/abs/2309.17357 · HTML: https://mlsanity.com/paper/2309.17357/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 68 references that are in the map.
- First posted on arXiv 29 Sep 2023.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Local SGD Converges Fast and Communicates Little](https://mlsanity.com/paper/1805.09767/) (2018)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016)
- [Training Neural Networks with Local Error Signals](https://mlsanity.com/paper/1901.06656/) (2019)
- [Maximum Mean Discrepancy Gradient Flow](https://mlsanity.com/paper/1906.04370/) (2019)
- [Greedy Layerwise Learning Can Scale to ImageNet](https://mlsanity.com/paper/1812.11446/) (2018)
- [Putting An End to End-to-End: Gradient-Isolated Learning of Representations](https://mlsanity.com/paper/1905.11786/) (2019)
- [Learning One-hidden-layer Neural Networks with Landscape Design](https://mlsanity.com/paper/1711.00501/) (2017)
- [Decoupled Greedy Learning of CNNs](https://mlsanity.com/paper/1901.08164/) (2019)
- [Residual Connections Encourage Iterative Inference](https://mlsanity.com/paper/1710.04773/) (2017)
- [Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diffusions](https://mlsanity.com/paper/1806.08141/) (2018)
- [The HSIC Bottleneck: Deep Learning without Back-Propagation](https://mlsanity.com/paper/1908.01580/) (2019)
- [Revisiting Locally Supervised Learning: an Alternative to End-to-end Training](https://mlsanity.com/paper/2101.10832/) (2021)
- [Refining Deep Generative Models via Discriminator Gradient Flow](https://mlsanity.com/paper/2012.00780/) (2020)
- [Decoupled Parallel Backpropagation with Convergence Guarantee](https://mlsanity.com/paper/1804.10574/) (2018)
- [Understanding Synthetic Gradients and Decoupled Neural Interfaces](https://mlsanity.com/paper/1703.00522/) (2017)
- [Learning Deep ResNet Blocks Sequentially using Boosting Theory](https://mlsanity.com/paper/1706.04964/) (2017)
- [Deep Generative Learning via Variational Gradient Flow](https://mlsanity.com/paper/1901.08469/) (2019)
- [Training Neural Networks Using Features Replay](https://mlsanity.com/paper/1807.04511/) (2018)
- [Dataset Dynamics via Gradient Flows in Probability Space](https://mlsanity.com/paper/2010.12760/) (2020)

## Built on by (most cited first, 4 of 4)

- [Scalable Model Merging with Progressive Layer-wise Distillation](https://mlsanity.com/paper/2502.12706/) (2025)
- [LaCoOT: Layer Collapse through Optimal Transport](https://mlsanity.com/paper/2406.08933/) (2024)
- [Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognition](https://mlsanity.com/paper/2510.27651/) (2025)
- [Depth-Wise Representation Development Under Blockwise Self-Supervised Learning for Video Vision Transformers](https://mlsanity.com/paper/2601.09040/) (2026)

## Abstract (arXiv)

Greedy layer-wise or module-wise training of neural networks is compelling in constrained and on-device settings where memory is limited, as it circumvents a number of problems of end-to-end back-propagation. However, it suffers from a stagnation problem, whereby early layers overfit and deeper layers stop increasing the test accuracy after a certain depth. We propose to solve this issue by introducing a module-wise regularization inspired by the minimizing movement scheme for gradient flows in distribution space. We call the method TRGL for Transport Regularized Greedy Learning and study it theoretically, proving that it leads to greedy modules that are regular and that progressively solve the task. Experimentally, we show improved accuracy of module-wise training of various architectures such as ResNets, Transformers and VGG, when our regularization is added, superior to that of other module-wise training methods and often to end-to-end training, with as much as 60% less memory usage.