# Efficient Training of Large Language Models on Distributed Infrastructures: A Survey

arXiv: https://arxiv.org/abs/2407.20018 · HTML: https://mlsanity.com/paper/2407.20018/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jul 2024.

## Built on by (most cited first, 27 of 27)

- [Adaptive Fault Tolerance Mechanisms of Large Language Models in Cloud Computing Environments](https://mlsanity.com/paper/2503.12228/) (2025)
- [A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation](https://mlsanity.com/paper/2512.06547/) (2025)
- [DeMo: Decoupled Momentum Optimization](https://mlsanity.com/paper/2411.19870/) (2024)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2503.16585/) (2025)
- [Direct Quantized Training of Language Models with Stochastic Rounding](https://mlsanity.com/paper/2412.04787/) (2024)
- [NoLoCo: No-all-reduce Low Communication Training Method for Large Models](https://mlsanity.com/paper/2506.10911/) (2025)
- [A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Design](https://mlsanity.com/paper/2607.09084/) (2026)
- [Scaling Laws of Graph Neural Networks for Atomistic Materials Modeling](https://mlsanity.com/paper/2504.08112/) (2025)
- [xLLM Technical Report](https://mlsanity.com/paper/2510.14686/) (2025)
- [Echo: Simulating Distributed Training At Scale](https://mlsanity.com/paper/2412.12487/) (2024)
- [Intelligent Orchestration of Distributed Large Foundation Model Inference at the Edge](https://mlsanity.com/paper/2504.03668/) (2025)
- [STAlloc: Enhancing Memory Efficiency in Large-Scale Model Training with Spatio-Temporal Planning](https://mlsanity.com/paper/2507.16274/) (2025)
- [FlashRecovery: Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMs](https://mlsanity.com/paper/2509.03047/) (2025)
- [Joint Partitioning and Placement of Foundation Models for Real-Time Edge AI](https://mlsanity.com/paper/2512.01039/) (2025)
- [Revisiting Parameter Server in LLM Post-Training](https://mlsanity.com/paper/2601.19362/) (2026)
- [FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic](https://mlsanity.com/paper/2510.24061/) (2025)
- [Crypto x AI, AI x Crypto: A Survey](https://mlsanity.com/paper/2606.13892/) (2026)
- [LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training](https://mlsanity.com/paper/2608.11919/) (2026)
- [Topology-aware Preemptive Scheduling for Co-located LLM Workloads](https://mlsanity.com/paper/2411.11560/) (2024)
- [MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall](https://mlsanity.com/paper/2509.02480/) (2025)
- [Horizon-LM: A RAM-Centric Architecture for LLM Training](https://mlsanity.com/paper/2602.04816/) (2026)
- [Scaling Vision Transformers: Evaluating DeepSpeed for Image-Centric Workloads](https://mlsanity.com/paper/2602.21081/) (2026)
- [Hybrid JIT-CUDA Graph Optimization for Low-Latency Large Language Model Inference](https://mlsanity.com/paper/2604.23467/) (2026)
- [CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training](https://mlsanity.com/paper/2604.24013/) (2026)
- [The Life of a Token: from Words to Bits on the Wire](https://mlsanity.com/paper/2609.19924/) (2026)
- [Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions](https://mlsanity.com/paper/2609.25463/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) like GPT and LLaMA are revolutionizing the AI industry with their sophisticated capabilities. Training these models requires vast GPU clusters and significant computing time, posing major challenges in terms of scalability, efficiency, and reliability. This survey explores recent advancements in training systems for LLMs, including innovations in training infrastructure with AI accelerators, networking, storage, and scheduling. Additionally, the survey covers parallelism strategies, as well as optimizations for computation, communication, and memory in distributed LLM training. It also includes approaches of maintaining system reliability over extended training periods. By examining current innovations and future directions, this survey aims to provide valuable insights towards improving LLM training systems and tackling ongoing challenges. Furthermore, traditional digital circuit-based computing systems face significant constraints in meeting the computational demands of LLMs, highlighting the need for innovative solutions such as optical computing and optical networks.