mlsanityData to 14 Oct 2026

APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving

arXiv 2411.17651 ↗First posted 26 Nov 2024cs.DCcited by 15

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Efficiently serving Large Language Models (LLMs) requires selecting an optimal parallel execution plan, balancing computation, memory, and communication overhead. However, determining the best strategy is challenging due to varying parallelism techniques (data, pipeline, tensor) and workload characteristics (e.g., compute-intensive tasks with long prompts vs. memory-intensive tasks with long generation). We propose APEX, an LLM serving system simulator that efficiently identifies optimal parallel execution plans by considering key factors of LLM serving systems, such as memory usage, batching behavior, etc. APEX performs dynamism-aware simulation to model iteration-level batching, and leverages LLMs' repetitive structure to reduce design space, scaling efficiently to trillion-scale models. APEX abstracts the key components of LLM serving systems, including the model, batching module, quantization formats, and device clusters, enabling the simulator to be general and extensible. Simulating on a CPU, APEX evaluates execution plans for various device clusters, covering diverse LLMs and workloads. APEX finds plans up to 3.37x faster than heuristics, and also plans that reduce energy consumption by up to 45% compared to latency-optimal plans. APEX performs comprehensive evaluations, reporting key system metrics like time per output token and time to first token, which can help service providers meet SLOs. APEX identifies an optimal plan within 15 minutes on a CPU, making it 71x faster and 1234x more cost-effective than cloud-based GPU deployment. APEX can be accessed at https://github.com/microsoft/apex_plus

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 52 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

34 of 52 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale10 Aug
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  4. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems23 Dec

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2024Mixtral of Expertscited by 1,823
  4. 2020GLU Variants Improve Transformercited by 1,804
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  10. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  11. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  12. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
18 more
  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  2. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  3. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 274
  4. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  5. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  6. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  7. 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 90
  8. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  9. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 73
  10. 2013Deep AutoRegressive Networkscited by 61
  11. 2024LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluationcited by 49
  12. 2019Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirectcited by 41
  13. 2023LLM4EDA: Emerging Progress in Large Language Models for Electronic Design Automationcited by 40
  14. 2021A Runtime-Based Computational Performance Predictor for Deep Neural Network Trainingcited by 29
  15. 2024Towards Next-Generation LLM-based Recommender Systems: A Survey and Beyondcited by 26
  16. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 23
  17. 2024LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Servicescited by 14
  18. 2023Proteus: Simulating the Performance of Distributed DNN Trainingcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 11.

15 papers.

Sort:

By area:

  1. 2026AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Servingcited by 9
  2. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 9
  3. 2025LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructurecited by 6
  4. 2026Revati: Transparent GPU-Free Time-Warp Emulation for LLM Servingcited by 5
  5. 2026Frontier: Towards Comprehensive and Accurate LLM Inference Simulationcited by 4
  6. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestrationcited by 3
  7. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  8. 2025P/D-Device: Disaggregated Large Language Model between Cloud and Devicescited by 2
  9. 2026KernelSight-LM: A Kernel-Level LLM Inference Simulatorcited by 2
  10. 2025Maya: Optimizing Deep Learning Training Workloads using GPU Runtime Emulationcited by 1
  11. 2025Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIMcited by 0
  12. 2026Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Servingcited by 0
3 more
  1. 2026LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Samplingcited by 0
  2. 2026Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulationcited by 0
  3. 2026Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  2. 2024Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels
  3. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  4. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughput
  5. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving
  6. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  7. 2026Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
  8. 2025Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling