# A Survey of LLM Inference Systems

arXiv: https://arxiv.org/abs/2506.21901 · HTML: https://mlsanity.com/paper/2506.21901/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jun 2025.

## Built on by (most cited first, 9 of 9)

- [Continuous Latent Diffusion Language Model](https://mlsanity.com/paper/2605.06548/) (2026)
- [Cognitive Load-Aware Inference: A Neuro-Symbolic Framework for Optimizing the Token Economy of Large Language Models](https://mlsanity.com/paper/2507.00653/) (2025)
- [The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project](https://mlsanity.com/paper/2603.21354/) (2026)
- [When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference](https://mlsanity.com/paper/2605.27435/) (2026)
- [OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency](https://mlsanity.com/paper/2511.22481/) (2025)
- [Thinking While Driving: A Concurrent Framework for Real-Time, LLM-Based Adaptive Routing](https://mlsanity.com/paper/2512.10610/) (2025)
- [Black-Box Inference of LLM Architectural Properties with Restrictive API Access](https://mlsanity.com/paper/2607.01313/) (2026)
- [LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs](https://mlsanity.com/paper/2608.03036/) (2026)
- [Composable CXL Memory as a Kubernetes-Native Shared Memory for LLM Serving](https://mlsanity.com/paper/2609.10790/) (2026)

## Abstract (arXiv)

The past few years has witnessed specialized large language model (LLM) inference systems, such as vLLM, SGLang, Mooncake, and DeepFlow, alongside rapid LLM adoption via services like ChatGPT. Driving these system design efforts is the unique autoregressive nature of LLM request processing, motivating new techniques for achieving high performance while preserving high inference quality over high-volume and high-velocity workloads. While many of these techniques are discussed across the literature, they have not been analyzed under the framework of a complete inference system, nor have the systems themselves been analyzed and compared. In this survey, we review these techniques, starting from operators and algorithms for request processing, then moving on to techniques for model optimization and execution, including kernel design, batching, and scheduling, before ending with techniques for memory management, including paged memory, eviction and offloading techniques, quantization, and cache persistence. Through these discussions, we show that these techniques fundamentally rely on load prediction, adaptive mechanisms, and cost reduction in order to overcome the challenges introduced by autoregressive generation and achieve the goals of the system. We then discuss how these techniques can be combined to form single-replica and multi-replica inference systems, including disaggregated inference systems that offer more control over resource allocation and serverless systems that can be deployed over shared hardware infrastructure. We end with a discussion of remaining challenges.