mlsanityData to 7 Oct 2026

Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Training

arXiv 2504.09307 ↗First posted 12 Apr 2025cs.DC cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Training LLMs in distributed environments presents significant challenges due to the complexity of model execution, deployment systems, and the vast space of configurable strategies. Although various optimization techniques exist, achieving high efficiency in practice remains difficult. Accurate performance models that effectively characterize and predict a model's behavior are essential for guiding optimization efforts and system-level studies. We propose Lumos, a trace-driven performance modeling and estimation toolkit for large-scale LLM training, designed to accurately capture and predict the execution behaviors of modern LLMs. We evaluate Lumos on a production ML cluster with up to 512 NVIDIA H100 GPUs using various GPT-3 variants, demonstrating that it can replay execution time with an average error of just 3.3%, along with other runtime details, across different models and configurations. Additionally, we validate its ability to estimate performance for new setups from existing traces, facilitating efficient exploration of model and deployment configurations.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 30 of its 45 references that are in the map. First posted on arXiv 12 Apr 2025.

Builds on

30 of 45 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  10. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  11. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  12. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
18 more
  1. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  2. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  3. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  4. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
  5. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  6. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  7. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  8. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  9. 2018Mixed Precision Training of Convolutional Neural Networks using Integer Operationscited by 45
  10. 2021RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inferencecited by 35
  11. 2018TicTac: Accelerating Distributed Deep Learning with Communication Schedulingcited by 27
  12. 2020Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Trainingcited by 22
  13. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 22
  14. 2021Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Modelscited by 15
  15. 2022dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Trainingcited by 12
  16. 2022Building a Performance Model for Deep Learning Recommendation Model Training on GPUscited by 10
  17. 2022A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Modelscited by 9
  18. 2022FPGA-based AI Smart NICs for Scalable Distributed AI Training Systemscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Scalable Synthesis of distributed LLM workloads through Symbolic Tensor Graphscited by 1
  2. 2025RedunCut: Measurement-Driven Sampling and Accuracy Performance Modeling for Low-Cost Live Video Analyticscited by 0
  3. 2026MLCommons Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Tracescited by 0
  4. 2026EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimizationcited by 0
  5. 2026MLSYSIM: First-Principles Infrastructure Modeling for Machine Learning Systemscited by 0
  6. 2026The Life of a Token: from Words to Bits on the Wirecited by 0