mlsanityData to 14 Oct 2026

Module-wise Training of Neural Networks via the Minimizing Movement Scheme

arXiv 2309.17357 ↗First posted 29 Sep 2023cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Greedy layer-wise or module-wise training of neural networks is compelling in constrained and on-device settings where memory is limited, as it circumvents a number of problems of end-to-end back-propagation. However, it suffers from a stagnation problem, whereby early layers overfit and deeper layers stop increasing the test accuracy after a certain depth. We propose to solve this issue by introducing a module-wise regularization inspired by the minimizing movement scheme for gradient flows in distribution space. We call the method TRGL for Transport Regularized Greedy Learning and study it theoretically, proving that it leads to greedy modules that are regular and that progressively solve the task. Experimentally, we show improved accuracy of module-wise training of various architectures such as ResNets, Transformers and VGG, when our regularization is added, superior to that of other module-wise training methods and often to end-to-end training, with as much as 60% less memory usage.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 42 of its 68 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

42 of 68 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2016Wide Residual Networkscited by 3,070
  8. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  9. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  10. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  11. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  12. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 257
30 more
  1. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 137
  2. 2019Training Neural Networks with Local Error Signalscited by 136
  3. 2019Maximum Mean Discrepancy Gradient Flowcited by 127
  4. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 115
  5. 2019Putting An End to End-to-End: Gradient-Isolated Learning of Representationscited by 86
  6. 2017Learning One-hidden-layer Neural Networks with Landscape Designcited by 81
  7. 2019Decoupled Greedy Learning of CNNscited by 78
  8. 2017Residual Connections Encourage Iterative Inferencecited by 77
  9. 2018Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diffusionscited by 77
  10. 2019The HSIC Bottleneck: Deep Learning without Back-Propagationcited by 77
  11. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  12. 2020Refining Deep Generative Models via Discriminator Gradient Flowcited by 45
  13. 2018Decoupled Parallel Backpropagation with Convergence Guaranteecited by 39
  14. 2017Understanding Synthetic Gradients and Decoupled Neural Interfacescited by 35
  15. 2017Learning Deep ResNet Blocks Sequentially using Boosting Theorycited by 32
  16. 2019Deep Generative Learning via Variational Gradient Flowcited by 32
  17. 2018Training Neural Networks Using Features Replaycited by 23
  18. 2020Dataset Dynamics via Gradient Flows in Probability Spacecited by 18
  19. 2021ResIST: Layer-Wise Decomposition of ResNets for Distributed Trainingcited by 18
  20. 2017Layer-wise training of deep networks using kernel similaritycited by 15
  21. 2019Towards Robust ResNet: A Small Step but A Giant Leapcited by 15
  22. 2022Gigapixel Whole-Slide Images Classification using Locally Supervised Learningcited by 15
  23. 2020A Principle of Least Action for the Training of Neural Networkscited by 8
  24. 2020Layer-wise training convolutional neural networks with smaller filters for human activity recognition using wearable sensorscited by 7
  25. 2018Improved Learning of One-hidden-layer Convolutional Neural Networks with Overlapscited by 6
  26. 2019Optimal Unsupervised Domain Translationcited by 6
  27. 2022Local Learning on Transformers via Feature Reconstructioncited by 4
  28. 2019On Residual Networks Learning a Perturbation from Identitycited by 2
  29. 2020A More Biologically Plausible Local Learning Rule for ANNscited by 2
  30. 2021Layer-Parallel Training of Residual Networks with Auxiliary-Variable Networkscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 10
  2. 2024LaCoOT: Layer Collapse through Optimal Transportcited by 1
  3. 2025Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognitioncited by 0
  4. 2026Depth-Wise Representation Development Under Blockwise Self-Supervised Learning for Video Vision Transformerscited by 0