mlsanityData to 14 Oct 2026

LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems

arXiv 2507.21276 ↗First posted 28 Jul 2025cs.AI cs.CL cs.DCcited by 3

Not on the map. Why?

Abstract (arXiv)

Modern deployment of large language models (LLMs) frequently involves both inference serving and continuous retraining to stay aligned with evolving data and user feedback. Common practices separate these workloads onto distinct servers in isolated phases, causing substantial inefficiencies (e.g., GPU idleness) and delayed adaptation to new data in distributed settings. Our empirical analysis reveals that these inefficiencies stem from dynamic request arrivals during serving and workload heterogeneity in pipeline-parallel training. To address these challenges, we propose LeMix, a system for co-locating and managing concurrent LLM serving and training workloads. LeMix integrates offline profiling, execution prediction mechanisms, and runtime scheduling to dynamically adapt resource allocation based on workload characteristics and system conditions. By understanding task-specific behaviors and co-execution interference across shared nodes, LeMix improves utilization and serving quality without compromising serving responsiveness. Our evaluation shows that LeMix improves throughput by up to 3.53x, reduces inference loss by up to 0.61x, and delivers up to 2.12x higher response time SLO attainment over traditional separate setups. To our knowledge, this is the first work to uncover and exploit the opportunities of joint LLM inference and training, paving the way for more resource-efficient deployment of LLMs in production environments.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 50 of its 75 references that are in the map. First posted on arXiv 28 Jul 2025.

Builds on

50 of 75 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
38 more
  1. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  2. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  3. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  7. 2022Efficiently Scaling Transformer Inferencecited by 539
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  9. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  11. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  12. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  13. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  14. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  15. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  16. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80
  17. 2023Fairness in Serving Large Language Modelscited by 78
  18. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  19. 2020Ekya: Continuous Learning of Video Analytics Models on Edge Compute Serverscited by 58
  20. 2023Large Language Models Empowered Autonomous Edge AI for Connected Intelligencecited by 56
  21. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  22. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  23. 2022Aryl: An Elastic Cluster Scheduler for Deep Learningcited by 27
  24. 2022PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matricescited by 24
  25. 2023White-Box Multi-Objective Adversarial Attack on Dialogue Generationcited by 14
  26. 2017AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networkscited by 12
  27. 2022Learning to Reverse DNNs from AI Programs Automaticallycited by 12
  28. 2022Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Modelscited by 11
  29. 2023RT-LM: Uncertainty-Aware Resource Management for Real-Time Inference of Language Modelscited by 11
  30. 2023Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attackcited by 11
  31. 2023DyCL: Dynamic Neural Network Compilation Via Program Rewriting and Graph Optimizationcited by 9
  32. 2023Uncertainty-Aware Bootstrap Learning for Joint Extraction on Distantly-Supervised Datacited by 7
  33. 2021Estimating Predictive Uncertainty Under Program Data Distribution Shiftcited by 5
  34. 2021Multi-model Machine Learning Inference Serving with GPU Spatial Partitioningcited by 4
  35. 2024Uncertainty Awareness of Large Language Models Under Code Distribution Shifts: A Benchmark Studycited by 3
  36. 2025Dr Genre: Reinforcement Learning from Decoupled LLM Feedback for Generic Text Rewritingcited by 3
  37. 2023Distantly-Supervised Joint Extraction with Noise-Robust Learningcited by 2
  38. 2025Mixtraining: A Better Trade-Off Between Compute and Performancecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025MACE: A Hybrid LLM Serving System with Colocated SLO-aware Continuous Retraining Alignmentcited by 2
  2. 2025Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networkscited by 1
  3. 2026PIMbot: A Self-Adaptive Attack Framework for Adversarial Manipulation of Multi-Robot Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  2. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  3. 2025CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancing
  4. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
  5. 2026KernelSight-LM: A Kernel-Level LLM Inference Simulator
  6. 2025APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs
  7. 2025Harli: SLO-Aware Co-location of LLM Inference and PEFT-based Finetuning on Model-as-a-Service Platforms
  8. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments