mlsanityData to 14 Oct 2026

Decoupled Parallel Backpropagation with Convergence Guarantee

arXiv 1804.10574 ↗First posted 27 Apr 2018cs.LG stat.MLcited by 39

Not on the map. Why?

Abstract (arXiv)

Backpropagation algorithm is indispensable for the training of feedforward neural networks. It requires propagating error gradients sequentially from the output layer all the way back to the input layer. The backward locking in backpropagation algorithm constrains us from updating network layers in parallel and fully leveraging the computing resources. Recently, several algorithms have been proposed for breaking the backward locking. However, their performances degrade seriously when networks are deep. In this paper, we propose decoupled parallel backpropagation algorithm for deep learning optimization with convergence guarantee. Firstly, we decouple the backpropagation algorithm using delayed gradients, and show that the backward locking is removed when we split the networks into multiple modules. Then, we utilize decoupled parallel backpropagation in two stochastic methods and prove that our method guarantees convergence to critical points for the non-convex problem. Finally, we perform experiments for training deep convolutional neural networks on benchmark datasets. The experimental results not only confirm our theoretical analysis, but also demonstrate that the proposed method can achieve significant speedup without loss of accuracy.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 8; 2023: 11; 2024: 7; 2025: 1; 2026: 5.

39 papers.

Sort:

By area:

  1. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 78
  2. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  3. 2021FLOP: Federated Learning on Medical Datasets using Partial Networkscited by 42
  4. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  5. 2022Efficient Attribute Unlearning: Towards Selective Removal of Input Attributes from Feature Representationscited by 17
  6. 2022Deep Incubation: Training Large Models by Divide-and-Conqueringcited by 13
  7. 2023The Trifecta: Three simple techniques for training deeper Forward-Forward networkscited by 12
  8. 2024F-OAL: Forward-only Online Analytic Learning with Fast Training and Low Memory Footprint in Class Incremental Learningcited by 12
  9. 2024Scaling Supervised Local Learning with Augmented Auxiliary Networkscited by 9
  10. 2022Low-Variance Forward Gradients using Direct Feedback Alignment and Momentumcited by 7
  11. 2021Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learningcited by 6
  12. 2022Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Serverscited by 6
27 more
  1. 2023Staleness-Alleviated Distributed GNN Training via Online Dynamic-Embedding Predictioncited by 6
  2. 2024PETRA: Parallel End-to-end Training with Reversible Architecturescited by 6
  3. 2023CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Modelcited by 5
  4. 2024MLAAN: Scaling Supervised Local Learning with Multilaminar Leap Augmented Auxiliary Networkcited by 5
  5. 2024HPFF: Hierarchical Locally Supervised Learning with Patch Feature Fusioncited by 5
  6. 2023Module-wise Training of Neural Networks via the Minimizing Movement Schemecited by 4
  7. 2021Incremental Layer-wise Self-Supervised Learning for Efficient Speech Domain Adaptation On Devicecited by 3
  8. 2021Enabling Large Batch Size Training for DNN Models Beyond the Memory Limit While Maintaining Performancecited by 3
  9. 2022GLEAM: Greedy Learning for Large-Scale Accelerated MRI Reconstructioncited by 3
  10. 2023Speed Up Federated Learning in Heterogeneous Environment: A Dynamic Tiering Approachcited by 3
  11. 2021Layer-Parallel Training of Residual Networks with Auxiliary-Variable Networkscited by 2
  12. 2022Layer-Wise Partitioning and Merging for Efficient and Scalable Deep Learningcited by 2
  13. 2026Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotationcited by 2
  14. 2022BackLink: Supervised Local Training with Backward Linkscited by 1
  15. 2022Block-wise Training of Residual Networks via the Minimizing Movement Schemecited by 1
  16. 2023ADA-GP: Accelerating DNN Training By Adaptive Gradient Predictioncited by 1
  17. 2021Front Contribution instead of Back Propagationcited by 0
  18. 2023Random Feedback Alignment Algorithms to train Neural Networks: Why do they Align?cited by 0
  19. 2023A Survey From Distributed Machine Learning to Distributed Deep Learningcited by 0
  20. 2023Go beyond End-to-End Training: Boosting Greedy Local Learning with Context Supplycited by 0
  21. 2024Forward Direct Feedback Alignment for Online Gradient Estimates of Spiking Neural Networkscited by 0
  22. 2024On the Convergence Theory of Pipeline Gradient-based Analog In-memory Trainingcited by 0
  23. 2025Distributed Deep Learning using Stochastic Gradient Stalenesscited by 0
  24. 2026Is Backpropagation Optimal? When Synthetic Gradients Improve Sample Efficiencycited by 0
  25. 2026Forward-Only Convolutional Neural Networks with Learnable Channel-Class Assignmentcited by 0
  26. 2026ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decouplingcited by 0
  27. 2026SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learningcited by 0