mlsanityData to 14 Oct 2026

Robust LLM Training Infrastructure at ByteDance

arXiv 2509.16293 ↗First posted 19 Sep 2025cs.LG cs.AI cs.DCcited by 16

Not on the map. Why?

Abstract (arXiv)

The training scale of large language models (LLMs) has reached tens of thousands of GPUs and is still continuously expanding, enabling faster learning of larger models. Accompanying the expansion of the resource scale is the prevalence of failures (CUDA error, NaN values, job hang, etc.), which poses significant challenges to training stability. Any large-scale LLM training infrastructure should strive for minimal training interruption, efficient fault diagnosis, and effective failure tolerance to enable highly efficient continuous training. This paper presents ByteRobust, a large-scale GPU infrastructure management system tailored for robust and stable training of LLMs. It exploits the uniqueness of LLM training process and gives top priorities to detecting and recovering failures in a routine manner. Leveraging parallelisms and characteristics of LLM training, ByteRobust enables high-capacity fault tolerance, prompt fault demarcation, and localization with an effective data-driven approach, comprehensively ensuring continuous and efficient training of LLM tasks. ByteRobust is deployed on a production GPU platform and achieves 97% ETTR for a three-month training job on 9,600 GPUs.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 42 of its 84 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

42 of 84 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
30 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  2. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  3. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  4. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  5. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  7. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  8. 2025Seed1.5-VL Technical Reportcited by 373
  9. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  10. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  11. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  12. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  13. 2024Large Language Models and Games: A Survey and Roadmapcited by 93
  14. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 82
  15. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  16. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 57
  17. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  18. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  19. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  20. 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
  21. 2024Revisiting Reliability in Large-Scale Machine Learning Research Clusterscited by 28
  22. 2024ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Developmentcited by 24
  23. 2021Silent Data Corruptions at Scalecited by 23
  24. 2024Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instancescited by 17
  25. 2023Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collectionscited by 16
  26. 2023Unicron: Economizing Self-Healing LLM Training at Scalecited by 16
  27. 2025Understanding Silent Data Corruption in LLM Trainingcited by 15
  28. 2024SuperBench: Improving Cloud AI Infrastructure Reliability with Proactive Validationcited by 11
  29. 2023SWIFT: Expedited Failure Recovery for Large-scale DNN Trainingcited by 5
  30. 2025MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Trainingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 12.

16 papers.

Sort:

By area:

  1. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
  2. 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Conditioncited by 23
  3. 2025RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMscited by 11
  4. 2026Training LLMs with Fault Tolerant HSDP on 100,000 GPUscited by 6
  5. 2025xLLM Technical Reportcited by 3
  6. 2026Exploring Silent Data Corruption as a Reliability Challenge in LLM Trainingcited by 3
  7. 2026GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Servingcited by 3
  8. 2025Role-Based Fault Tolerance System for LLM RL Post-Trainingcited by 2
  9. 2026DataStates-LLM: Scalable Checkpointing for Transformer Models Using Composable State Providerscited by 1
  10. 2026ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workloadcited by 0
  11. 2026A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with PrismLLMcited by 0
  12. 2026SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Trainingcited by 0
4 more
  1. 2026TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Trainingcited by 0
  2. 2026Leto: Fast In-Place Recovery for LLM Training on Surviving Hardwarecited by 0
  3. 2026Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelinescited by 0
  4. 2026TSRL: Trigger-based Supervised Reinforcement Learning for Continual Agent Trainingcited by 0