mlsanityData to 7 Oct 2026

Block-wise Training of Residual Networks via the Minimizing Movement Scheme

arXiv 2210.00949 ↗First posted 3 Oct 2022cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

End-to-end backpropagation has a few shortcomings: it requires loading the entire model during training, which can be impossible in constrained settings, and suffers from three locking problems (forward locking, update locking and backward locking), which prohibit training the layers in parallel. Solving layer-wise optimization problems can address these problems and has been used in on-device training of neural networks. We develop a layer-wise training method, particularly welladapted to ResNets, inspired by the minimizing movement scheme for gradient flows in distribution space. The method amounts to a kinetic energy regularization of each block that makes the blocks optimal transport maps and endows them with regularity. It works by alleviating the stagnation problem observed in layer-wise training, whereby greedily-trained early layers overfit and deeper layers stop increasing test accuracy after a certain depth. We show on classification tasks that the test accuracy of block-wise trained ResNets is improved when using our method, whether the blocks are trained sequentially or in parallel.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 40 of its 57 references that are in the map. First posted on arXiv 3 Oct 2022.

Builds on

40 of 57 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Ordinary Differential Equationscited by 3,733
  3. 2016Wide Residual Networkscited by 3,057
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  6. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  8. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  9. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  10. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  11. 2020How to train your neural ODE: the world of Jacobian and kinetic regularizationcited by 228
  12. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
28 more
  1. 2018To understand deep learning we need to understand kernel learningcited by 186
  2. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 135
  3. 2019Training Neural Networks with Local Error Signalscited by 135
  4. 2019Maximum Mean Discrepancy Gradient Flowcited by 123
  5. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 113
  6. 2019Putting An End to End-to-End: Gradient-Isolated Learning of Representationscited by 85
  7. 2018Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diffusionscited by 77
  8. 2019The HSIC Bottleneck: Deep Learning without Back-Propagationcited by 77
  9. 2019Decoupled Greedy Learning of CNNscited by 76
  10. 2017Residual Connections Encourage Iterative Inferencecited by 74
  11. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  12. 2020LoCo: Local Contrastive Representation Learningcited by 55
  13. 2020Refining Deep Generative Models via Discriminator Gradient Flowcited by 45
  14. 2017Deep supervised learning using local errorscited by 41
  15. 2018Decoupled Parallel Backpropagation with Convergence Guaranteecited by 39
  16. 2017Understanding Synthetic Gradients and Decoupled Neural Interfacescited by 35
  17. 2017Learning Deep ResNet Blocks Sequentially using Boosting Theorycited by 32
  18. 2019Deep Generative Learning via Variational Gradient Flowcited by 32
  19. 2018Training Neural Networks Using Features Replaycited by 23
  20. 2020Dataset Dynamics via Gradient Flows in Probability Spacecited by 18
  21. 2021ResIST: Layer-Wise Decomposition of ResNets for Distributed Trainingcited by 18
  22. 2017Layer-wise training of deep networks using kernel similaritycited by 15
  23. 2019Towards Robust ResNet: A Small Step but A Giant Leapcited by 15
  24. 2020A Principle of Least Action for the Training of Neural Networkscited by 8
  25. 2020Layer-wise training convolutional neural networks with smaller filters for human activity recognition using wearable sensorscited by 7
  26. 2019Optimal Unsupervised Domain Translationcited by 6
  27. 2019On Residual Networks Learning a Perturbation from Identitycited by 2
  28. 2020A More Biologically Plausible Local Learning Rule for ANNscited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022Local Learning on Transformers via Feature Reconstructioncited by 4