# APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving

arXiv: https://arxiv.org/abs/2411.17651 · HTML: https://mlsanity.com/paper/2411.17651/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 34 of its 52 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 30 of 34)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://mlsanity.com/paper/2310.01889/) (2023)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://mlsanity.com/paper/2304.01433/) (2023)
- [Vidur: A Large-Scale Simulation Framework For LLM Inference](https://mlsanity.com/paper/2405.05465/) (2024)
- [DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency](https://mlsanity.com/paper/2408.00741/) (2024)
- [Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems](https://mlsanity.com/paper/2312.15234/) (2023)
- [PyTorch Distributed: Experiences on Accelerating Data Parallel Training](https://mlsanity.com/paper/2006.15704/) (2020)
- [Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities](https://mlsanity.com/paper/2405.10825/) (2024)
- [Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation](https://mlsanity.com/paper/2308.10335/) (2023)
- [Deep AutoRegressive Networks](https://mlsanity.com/paper/1310.8499/) (2013)
- [LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation](https://mlsanity.com/paper/2404.10100/) (2024)
- [Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect](https://mlsanity.com/paper/1903.04611/) (2019)
- [LLM4EDA: Emerging Progress in Large Language Models for Electronic Design Automation](https://mlsanity.com/paper/2401.12224/) (2023)
- [LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale](https://mlsanity.com/paper/2408.05499/) (2024)
- [A Runtime-Based Computational Performance Predictor for Deep Neural Network Training](https://mlsanity.com/paper/2102.00527/) (2021)

## Built on by (most cited first, 15 of 15)

- [AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving](https://mlsanity.com/paper/2601.06288/) (2026)
- [LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure](https://mlsanity.com/paper/2602.23036/) (2026)
- [LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructure](https://mlsanity.com/paper/2511.07229/) (2025)
- [Revati: Transparent GPU-Free Time-Warp Emulation for LLM Serving](https://mlsanity.com/paper/2601.00397/) (2026)
- [Frontier: Towards Comprehensive and Accurate LLM Inference Simulation](https://mlsanity.com/paper/2605.21312/) (2026)
- [OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration](https://mlsanity.com/paper/2602.12151/) (2026)
- [Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulation](https://mlsanity.com/paper/2606.01725/) (2026)
- [P/D-Device: Disaggregated Large Language Model between Cloud and Devices](https://mlsanity.com/paper/2508.09035/) (2025)
- [KernelSight-LM: A Kernel-Level LLM Inference Simulator](https://mlsanity.com/paper/2606.28565/) (2026)
- [Maya: Optimizing Deep Learning Training Workloads using GPU Runtime Emulation](https://mlsanity.com/paper/2503.20191/) (2025)
- [Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM](https://mlsanity.com/paper/2510.05632/) (2025)
- [Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving](https://mlsanity.com/paper/2605.00254/) (2026)
- [LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Sampling](https://mlsanity.com/paper/2605.00616/) (2026)
- [Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation](https://mlsanity.com/paper/2605.07985/) (2026)
- [Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systems](https://mlsanity.com/paper/2608.24650/) (2026)

## Abstract (arXiv)

Efficiently serving Large Language Models (LLMs) requires selecting an optimal parallel execution plan, balancing computation, memory, and communication overhead. However, determining the best strategy is challenging due to varying parallelism techniques (data, pipeline, tensor) and workload characteristics (e.g., compute-intensive tasks with long prompts vs. memory-intensive tasks with long generation). We propose APEX, an LLM serving system simulator that efficiently identifies optimal parallel execution plans by considering key factors of LLM serving systems, such as memory usage, batching behavior, etc. APEX performs dynamism-aware simulation to model iteration-level batching, and leverages LLMs' repetitive structure to reduce design space, scaling efficiently to trillion-scale models. APEX abstracts the key components of LLM serving systems, including the model, batching module, quantization formats, and device clusters, enabling the simulator to be general and extensible. Simulating on a CPU, APEX evaluates execution plans for various device clusters, covering diverse LLMs and workloads. APEX finds plans up to 3.37x faster than heuristics, and also plans that reduce energy consumption by up to 45% compared to latency-optimal plans. APEX performs comprehensive evaluations, reporting key system metrics like time per output token and time to first token, which can help service providers meet SLOs. APEX identifies an optimal plan within 15 minutes on a CPU, making it 71x faster and 1234x more cost-effective than cloud-based GPU deployment. APEX can be accessed at https://github.com/microsoft/apex_plus