mlsanityData to 7 Oct 2026

LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure

arXiv 2602.23036 ↗First posted 26 Feb 2026cs.DC cs.AIcited by 8

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language model (LLM) serving infrastructures are undergoing a shift toward heterogeneity and disaggregation. Modern deployments increasingly integrate diverse accelerators and near-memory processing technologies, introducing significant hardware heterogeneity, while system software increasingly separates computation, memory, and model components across distributed resources to improve scalability and efficiency. As a result, LLM serving performance is no longer determined by hardware or software choices in isolation, but by their runtime interaction through scheduling, data movement, and interconnect behavior. However, understanding these interactions remains challenging, as existing simulators lack the ability to jointly model heterogeneous hardware and disaggregated serving techniques within a unified, runtime-driven framework. This paper presents LLMServingSim 2.0, a unified system-level simulator designed to make runtime-driven hardware-software interactions in heterogeneous and disaggregated LLM serving infrastructures explicit and analyzable. LLMServingSim 2.0 embeds serving decisions and hardware behavior into a single runtime loop, enabling interaction-aware modeling of batching, routing, offloading, memory, and power. The simulator supports extensible integration of emerging accelerators and memory systems through profile-based modeling, while capturing dynamic serving behavior and system-level effects. We validate LLMServingSim 2.0 against real deployments, showing that it reproduces key performance, memory, and power metrics with an average error of 0.95%, while maintaining simulation times of around 10 minutes even for complex configurations. These results demonstrate that LLMServingSim 2.0 provides a practical bridge between hardware innovation and serving-system design, enabling systematic exploration and co-design for next-generation LLM serving infrastructures.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 62 references that are in the map. First posted on arXiv 26 Feb 2026.

Builds on

36 of 62 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitiveness12 Sep
  2. 2025TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systems11 Mar
  3. 2025ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughput6 Mar
  4. 2024APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving26 Nov
  5. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scale15 Aug
  6. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale10 Aug
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  8. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov

Other references · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  7. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  8. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  9. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  10. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  11. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  12. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
16 more
  1. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  2. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  3. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  4. 2024NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencingcited by 48
  5. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  6. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 46
  7. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  8. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 36
  9. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batchingcited by 33
  10. 2023AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingcited by 31
  11. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 22
  12. 2025PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing Systemcited by 19
  13. 2024SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Expertscited by 18
  14. 2025DeepServe: Serverless Large Language Model Serving at Scalecited by 17
  15. 2023Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Tracescited by 13
  16. 2025CXL-GPU: Pushing GPU Memory Boundaries with the Integration of CXL Technologiescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Frontier: Towards Comprehensive and Accurate LLM Inference Simulationcited by 4
  2. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  3. 2026AgentServeSim: Serving-System Simulation and Policy Search for LLM Agent Programscited by 1
  4. 2026EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimizationcited by 0
  5. 2026ASTRA-sim 3.0: Next-Level Distributed Machine Learning Simulations via High-Fidelity GPU and Infrastructure Modelingcited by 0
  6. 2026Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systemscited by 0
  7. 2026Characterizing High Bandwidth Flash for LLM Servingcited by 0
  8. 2026Leto: Fast In-Place Recovery for LLM Training on Surviving Hardwarecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructure
  2. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  3. 2025Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM
  4. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
  5. 2026LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs
  6. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
  7. 2025RAPID-LLM: Resilience-Aware Performance analysis of Infrastructure for Distributed LLM Training and Inference ↗
  8. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving