mlsanityData to 7 Oct 2026

Local Critic Training for Model-Parallel Learning of Deep Neural Networks

arXiv 2102.01963 ↗First posted 3 Feb 2021cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a novel model-parallel learning method, called local critic training, which trains neural networks using additional modules called local critic networks. The main network is divided into several layer groups and each layer group is updated through error gradients estimated by the corresponding local critic network. We show that the proposed approach successfully decouples the update process of the layer groups for both convolutional neural networks (CNNs) and recurrent neural networks (RNNs). In addition, we demonstrate that the proposed method is guaranteed to converge to a critical point. We also show that trained networks by the proposed method can be used for structural optimization. Experimental results show that our method achieves satisfactory performance, reduces training time greatly, and decreases memory consumption per machine. Code is available at https://github.com/hjdw2/Local-critic-training.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 25 of its 53 references that are in the map. First posted on arXiv 3 Feb 2021.

Builds on

25 of 53 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  4. 2016YOLO9000: Better, Faster, Strongercited by 1,429
  5. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  6. 2017Mixed Precision Trainingcited by 1,135
  7. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  8. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  9. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  10. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  11. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  12. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 154
13 more
  1. 2017Sobolev Training for Neural Networkscited by 136
  2. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 135
  3. 2016Training Neural Networks Without Gradients: A Scalable ADMM Approachcited by 61
  4. 2016AdaNet: Adaptive Structural Learning of Artificial Neural Networkscited by 54
  5. 2019PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfigurationcited by 41
  6. 2018Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networkscited by 23
  7. 2018Training Neural Networks Using Features Replaycited by 23
  8. 2017Integrated Model, Batch and Domain Parallelism in Training Neural Networkscited by 16
  9. 2017AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networkscited by 12
  10. 2020TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelismcited by 10
  11. 2019Optimizing Multi-GPU Parallelization Strategies for Deep Learning Trainingcited by 9
  12. 2018Local Critic Training of Deep Neural Networkscited by 3
  13. 2018Data-parallel distributed training of very large models beyond GPU capacitycited by 2

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 1; 2023: 2; 2024: 1.

5 papers.

  1. 2021Single-Layer Vision Transformers for More Accurate Early Exits with Less Overheadcited by 26
  2. 2022BackLink: Supervised Local Training with Backward Linkscited by 1
  3. 2024Network Fission Ensembles for Low-Cost Self-Ensemblescited by 1
  4. 2023ADMM Algorithms for Residual Network Training: Convergence Analysis and Parallel Implementationcited by 0
  5. 2023LiPar: A Lightweight Parallel Learning Model for Practical In-Vehicle Network Intrusion Detectioncited by 0