mlsanityData to 14 Oct 2026

DHO2: Accelerating Distributed Hybrid Order Optimization via Model Parallelism and ADMM

arXiv 2505.00982 ↗First posted 2 May 2025cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Scaling deep neural network (DNN) training to more devices can reduce time-to-solution. However, it is impractical for users with limited computing resources. FOSI, as a hybrid order optimizer, converges faster than conventional optimizers by taking advantage of both gradient information and curvature information when updating the DNN model. Therefore, it provides a new chance for accelerating DNN training in the resource-constrained setting. In this paper, we explore its distributed design, namely DHO2, including distributed calculation of curvature information and model update with partial curvature information to accelerate DNN training with a low memory burden. To further reduce the training time, we design a novel strategy to parallelize the calculation of curvature information and the model update on different devices. Experimentally, our distributed design can achieve an approximate linear reduction of memory burden on each device with the increase of the device number. Meanwhile, it achieves 1.4×~2.1× speedup in the total training time compared with other distributed designs based on conventional first- and second-order optimizers.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 39 references that are in the map. First posted on arXiv 2 May 2025.

Builds on

19 of 39 references.

References · 19

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  7. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  8. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  9. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 334
  10. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  11. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 77
  12. 2015Preconditioned Stochastic Gradient Descentcited by 50
7 more
  1. 2022Federated Learning via Inexact ADMMcited by 44
  2. 2019Large-Batch Training for LSTM and Beyondcited by 21
  3. 2023Resfusion: Denoising Diffusion Probabilistic Models for Image Restoration Based on Prior Residual Noisecited by 19
  4. 20244-bit Shampoo for Memory-Efficient Network Trainingcited by 18
  5. 2025Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysiscited by 3
  6. 2025Preconditioned Inexact Stochastic ADMM for Deep Modelcited by 2
  7. 2023FOSI: Hybrid First and Second Order Optimizationcited by 1

Built on by

Not cited yet.