mlsanityData to 14 Oct 2026

DeInfoReg: A Decoupled Learning Framework for Better Training Throughput

arXiv 2506.18193 ↗First posted 22 Jun 2025cs.LG cs.AI cs.DCcited by 1

Not on the map. Why?

Abstract (arXiv)

This paper introduces Decoupled Supervised Learning with Information Regularization (DeInfoReg), a novel approach that transforms a long gradient flow into multiple shorter ones, thereby mitigating the vanishing gradient problem. Integrating a pipeline strategy, DeInfoReg enables model parallelization across multiple GPUs, significantly improving training throughput. We compare our proposed method with standard backpropagation and other gradient flow decomposition techniques. Extensive experiments on diverse tasks and datasets demonstrate that DeInfoReg achieves superior performance and better noise resistance than traditional BP models and efficiently utilizes parallel computing resources. The code for reproducibility is available at: https://github.com/ianzih/Decoupled-Supervised-Learning-for-Information-Regularization/.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 22 Jun 2025.

Builds on

28 of 41 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2014Going Deeper with Convolutionscited by 4,241
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2020Supervised Contrastive Learningcited by 2,829
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
16 more
  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  2. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  3. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  4. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  5. 2014Deeply-Supervised Netscited by 367
  6. 2019On Mutual Information Maximization for Representation Learningcited by 316
  7. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  8. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  9. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 137
  10. 2019Training Neural Networks with Local Error Signalscited by 136
  11. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  12. 2019Spatiotemporal Co-attention Recurrent Neural Networks for Human-Skeleton Motion Predictioncited by 47
  13. 2022Self-Supervised Learning with an Information Maximization Criterioncited by 38
  14. 2023Blockwise Self-Supervised Learning at Scalecited by 22
  15. 2024Towards Interpretable Deep Local Learning with Successive Gradient Reconciliationcited by 4
  16. 2019Associated Learning: Decomposing End-to-end Backpropagation based on Auto-encoders and Target Propagationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SCPL: Enhancing Neural Network Training Throughput with Decoupled Local Losses and Model Parallelismcited by 1