mlsanityData to 14 Oct 2026

Rethinking Memory and Communication Cost for Efficient Large Language Model Training

arXiv 2310.06003 ↗First posted 9 Oct 2023cs.LG cs.AIcited by 4

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Recently, various distributed strategies for large language model training have been proposed. However, these methods provided limited solutions for the trade-off between memory consumption and communication cost. In this paper, we rethink the impact of memory consumption and communication costs on the training speed of large language models, and propose a memory-communication balanced strategy set Partial Redundancy Optimizer (PaRO). PaRO provides comprehensive options which reduces the amount and frequency of inter-group communication with minor memory redundancy by fine-grained sharding strategy, thereby improving the training efficiency in various training scenarios. Additionally, we propose a Hierarchical Overlapping Ring (HO-Ring) communication topology to enhance communication efficiency between nodes or across switches in large language model training. Our experiments demonstrate that PaRO significantly improves training throughput by 1.19x-2.50x compared to the SOTA method and achieves a near-linear scalability. The HO-Ring algorithm improves communication efficiency by 36.5% compared to the traditional Ring algorithm.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 24 of its 35 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

24 of 35 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  9. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  10. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  12. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
12 more
  1. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  2. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  3. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 173
  4. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  5. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  6. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  7. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 52
  8. 2019Blink: Fast and Generic Collectives for Distributed MLcited by 36
  9. 20211-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speedcited by 24
  10. 2021Maximizing Parallelism in Distributed Training for Huge Neural Networkscited by 23
  11. 2022MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloudcited by 22
  12. 2018Improving all-reduce collective operations for imbalanced process arrival patternscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 1.

4 papers.

  1. 2023An Adaptive Placement and Parallelism Framework for Accelerating RLHF Trainingcited by 13
  2. 2025Efficient Distributed Optimization under Heavy-Tailed Noisecited by 7
  3. 2024M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretrainingcited by 4
  4. 2026Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Positioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025System-performance and cost modeling of Large Language Model training and inference
  2. 2025Overcoming the Communication-Performance Tradeoff in LLM Pretraining
  3. 2024Efficiency optimization of large-scale language models based on deep learning in natural language processing tasks
  4. 2026Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm
  5. 2024Search for Efficient Large Language Models
  6. 2025Scaling and Transferability of Annealing Strategies in Large Language Model Training
  7. 2026Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models
  8. 2024Hiding Communication Cost in Distributed LLM Training via Micro-batch Co-execution