mlsanityData to 7 Oct 2026

Enhancing Neural Training via a Correlated Dynamics Model

arXiv 2312.13247 ↗First posted 20 Dec 2023cs.LG math.DScited by 5

Not on the map. Why?

Abstract (arXiv)

As neural networks grow in scale, their training becomes both computationally demanding and rich in dynamics. Amidst the flourishing interest in these training dynamics, we present a novel observation: Parameters during training exhibit intrinsic correlations over time. Capitalizing on this, we introduce Correlation Mode Decomposition (CMD). This algorithm clusters the parameter space into groups, termed modes, that display synchronized behavior across epochs. This enables CMD to efficiently represent the training dynamics of complex networks, like ResNets and Transformers, using only a few modes. Moreover, test set generalization is enhanced. We introduce an efficient CMD variant, designed to run concurrently with training. Our experiments indicate that CMD surpasses the state-of-the-art method for compactly modeled dynamics on image classification. Our modeling can improve training efficiency and lower communication overhead, as shown by our preliminary experiments in the context of federated learning.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 39 of its 77 references that are in the map. First posted on arXiv 20 Dec 2023.

Builds on

39 of 77 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2016Wide Residual Networkscited by 3,057
  9. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  10. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  11. 2016Pyramid Scene Parsing Networkcited by 2,472
  12. 2016Identity Mappings in Deep Residual Networkscited by 1,960
27 more
  1. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  2. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  5. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  6. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,095
  7. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  8. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  9. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  10. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  11. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  12. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  13. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  14. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  15. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 55
  16. 2014A survey of dimensionality reduction techniquescited by 50
  17. 2020Optimizing Neural Networks via Koopman Operator Theorycited by 32
  18. 2018TicTac: Accelerating Distributed Deep Learning with Communication Schedulingcited by 27
  19. 2020Improving Neural Network Training in Low Dimensional Random Basescited by 19
  20. 2021An Operator Theoretic View on Pruning Deep Neural Networkscited by 18
  21. 2019Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activationcited by 14
  22. 2018MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithmscited by 12
  23. 2020Applications of Koopman Mode Analysis to Neural Networkscited by 12
  24. 2016Spectral Decompositions using One-Homogeneous Functionalscited by 7
  25. 2022GlueFL: Reconciling Client Sampling and Model Masking for Bandwidth Efficient Federated Learningcited by 7
  26. 2022Learning to Optimize with Dynamic Mode Decompositioncited by 5
  27. 2023Learning Linear Embeddings for Non-Linear Network Dynamics with Koopman Message Passingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Training-Free Policy Violation Detection via Activation-Space Whitening in LLMscited by 6
  2. 2026Distributional Spectral Diagnostics for Localizing Grokking Transitionscited by 2
  3. 2025Accelerating Learned Image Compression Through Modeling Neural Training Dynamicscited by 1
  4. 2025Measuring the Datacited by 0
  5. 2025Winning the Lottery by Preserving Network Training Dynamics with Concrete Ticket Searchcited by 0