mlsanityData to 7 Oct 2026

Efficient Training of Large Language Models on Distributed Infrastructures: A Survey

arXiv 2407.20018 ↗First posted 29 Jul 2024cs.DCcited by 26

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) like GPT and LLaMA are revolutionizing the AI industry with their sophisticated capabilities. Training these models requires vast GPU clusters and significant computing time, posing major challenges in terms of scalability, efficiency, and reliability. This survey explores recent advancements in training systems for LLMs, including innovations in training infrastructure with AI accelerators, networking, storage, and scheduling. Additionally, the survey covers parallelism strategies, as well as optimizations for computation, communication, and memory in distributed LLM training. It also includes approaches of maintaining system reliability over extended training periods. By examining current innovations and future directions, this survey aims to provide valuable insights towards improving LLM training systems and tackling ongoing challenges. Furthermore, traditional digital circuit-based computing systems face significant constraints in meeting the computational demands of LLMs, highlighting the need for innovative solutions such as optical computing and optical networks.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Jul 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 13; 2026: 9.

26 papers.

Sort:

By area:

  1. 2025Adaptive Fault Tolerance Mechanisms of Large Language Models in Cloud Computing Environmentscited by 12
  2. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  3. 2025A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximationcited by 11
  4. 2024DeMo: Decoupled Momentum Optimizationcited by 10
  5. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  6. 2024Direct Quantized Training of Language Models with Stochastic Roundingcited by 5
  7. 2025Scaling Laws of Graph Neural Networks for Atomistic Materials Modelingcited by 4
  8. 2025NoLoCo: No-all-reduce Low Communication Training Method for Large Modelscited by 4
  9. 2025xLLM Technical Reportcited by 3
  10. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 3
  11. 2024Echo: Simulating Distributed Training At Scalecited by 2
  12. 2025STAlloc: Enhancing Memory Efficiency in Large-Scale Model Training with Spatio-Temporal Planningcited by 2
14 more
  1. 2025FlashRecovery: Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMscited by 2
  2. 2026Revisiting Parameter Server in LLM Post-Trainingcited by 2
  3. 2025Intelligent Orchestration of Distributed Large Foundation Model Inference at the Edgecited by 1
  4. 2025FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmeticcited by 1
  5. 2025Joint Partitioning and Placement of Foundation Models for Real-Time Edge AIcited by 1
  6. 2026LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Trainingcited by 1
  7. 2024Topology-aware Preemptive Scheduling for Co-located LLM Workloadscited by 0
  8. 2025MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wallcited by 0
  9. 2026Horizon-LM: A RAM-Centric Architecture for LLM Trainingcited by 0
  10. 2026Scaling Vision Transformers: Evaluating DeepSpeed for Image-Centric Workloadscited by 0
  11. 2026Hybrid JIT-CUDA Graph Optimization for Low-Latency Large Language Model Inferencecited by 0
  12. 2026CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Trainingcited by 0
  13. 2026The Life of a Token: from Words to Bits on the Wirecited by 0
  14. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0