mlsanityData to 14 Oct 2026

ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference

arXiv 2404.07947 ↗First posted 15 Mar 2024cs.DC cs.LGcited by 45

Not on the map. Why?

Cited by 22 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents ExeGPT, a distributed system designed for constraint-aware LLM inference. ExeGPT finds and runs with an optimal execution schedule to maximize inference throughput while satisfying a given latency constraint. By leveraging the distribution of input and output sequences, it effectively allocates resources and determines optimal execution configurations, including batch sizes and partial tensor parallelism. We also introduce two scheduling strategies based on Round-Robin Allocation and Workload-Aware Allocation policies, suitable for different NLP workloads. We evaluate ExeGPT on six LLM instances of T5, OPT, and GPT-3 and five NLP tasks, each with four distinct latency constraints. Compared to FasterTransformer, ExeGPT achieves up to 15.2x improvements in throughput and 6x improvements in latency. Overall, ExeGPT achieves an average throughput gain of 2.9x across twenty evaluation scenarios. Moreover, when adapting to changing sequence distributions, the cost of adjusting the schedule in ExeGPT is reasonably modest. ExeGPT proves to be an effective solution for optimizing and executing LLM inference for diverse NLP workload and serving conditions.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 33 of its 52 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

33 of 52 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
21 more
  1. 2020GLU Variants Improve Transformercited by 1,804
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  7. 2015Teaching Machines to Read and Comprehendcited by 1,049
  8. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  9. 2018CoQA: A Conversational Question Answering Challengecited by 683
  10. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  12. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  13. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  14. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 156
  15. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  16. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  17. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  18. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 75
  19. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  20. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  21. 2021SciXGen: A Scientific Paper Dataset for Context-Aware Text Generationcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 23; 2026: 10.

45 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  2. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  3. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  4. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 31
  5. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooningcited by 24
  6. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 22
  7. 2025PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing Systemcited by 20
  8. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 19
  9. 2024Interactive Speculative Planning: Enhance Agent Efficiency through Co-design of System and User Interfacecited by 16
  10. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storagecited by 16
  11. 2025L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Modelscited by 15
  12. 2024Aladdin: Joint Placement and Scaling for SLO-Aware LLM Servingcited by 11
33 more
  1. 2024Leveraging Large Language Models for Integrated Satellite-Aerial-Terrestrial Networks: Recent Advances and Future Directionscited by 10
  2. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  3. 2024Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environmentscited by 9
  4. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictorcited by 9
  5. 2025Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Expertscited by 7
  6. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Servingcited by 6
  7. 2025FairBatching: Fairness-Aware Batch Formation for LLM Inferencecited by 6
  8. 2025MACKO: Sparse Matrix-Vector Multiplication for Low Sparsitycited by 6
  9. 2024PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inferencecited by 4
  10. 2025Ascendra: Dynamic Request Prioritization for Efficient LLM Servingcited by 4
  11. 2025Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latencycited by 4
  12. 2025CoDec: Prefix-Shared Decoding Kernel for LLMscited by 4
  13. 2025Attention to Non-Adopterscited by 4
  14. 2024MTP: A Meaning-Typed Language Abstraction for AI-Integrated Programmingcited by 3
  15. 2025AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applicationscited by 2
  16. 2025iServe: An Intent-based Serving System for LLMscited by 1
  17. 2025HERA: Hybrid Edge-cloud Resource Allocation for Cost-Efficient AI Agentscited by 1
  18. 2025FlexiQ: Adaptive Mixed-Precision Quantization for Latency/Accuracy Trade-Offs in Deep Neural Networkscited by 1
  19. 2025GoodSpeed: Optimizing Fair Goodput with Adaptive Speculative Decoding in Distributed Edge Inferencecited by 1
  20. 2026Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimatecited by 1
  21. 2026LMEdge: QoS-Aware LLM Inference Orchestration on Edge Clusterscited by 1
  22. 2024EconoServe: Maximizing Multi-Resource Utilization with SLO Guarantees in LLM Servingcited by 0
  23. 2024Software Performance Engineering for Foundation Model-Powered Softwarecited by 0
  24. 2025Mitigating KV Cache Competition to Enhance User Experience in LLM Inferencecited by 0
  25. 2025A Predictive and Synergistic Two-Layer Scheduling Framework for LLM Servingcited by 0
  26. 2026What Artificial Intelligence can do for High-Performance Computing systems?cited by 0
  27. 2026Tropical: Enhancing SLO Attainment in Disaggregated LLM Serving via SLO-Aware Multiplexingcited by 0
  28. 2026Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matricescited by 0
  29. 2026Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Schedulingcited by 0
  30. 2026Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attentioncited by 0
  31. 2026BALANCE: Hybrid Autoregressive-Speculative LLM Inference at the Network Edgecited by 0
  32. 2026DRLM: Deep Reinforcement Learning-Based LLM Query Orchestration in Edge Environmentscited by 0
  33. 2026Efficient Vision-Language-Action Management and Serving for Robot Factoriescited by 0