# Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads

arXiv: https://arxiv.org/abs/2401.11181 · HTML: https://mlsanity.com/paper/2401.11181/

## Key facts

- Cited by 121 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jan 2024.

## Built on by (most cited first, 30 of 121)

- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [A Survey on Efficient Inference for Large Language Models](https://mlsanity.com/paper/2404.14294/) (2024)
- [Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving](https://mlsanity.com/paper/2407.00079/) (2024)
- [A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond](https://mlsanity.com/paper/2503.21614/) (2025)
- [LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism](https://mlsanity.com/paper/2404.09526/) (2024)
- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention](https://mlsanity.com/paper/2405.04437/) (2024)
- [Efficient LLM Scheduling by Learning to Rank](https://mlsanity.com/paper/2408.15792/) (2024)
- [MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool](https://mlsanity.com/paper/2406.17565/) (2024)
- [EPIC: Efficient Position-Independent Caching for Serving Large Language Models](https://mlsanity.com/paper/2410.15332/) (2024)
- [POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference](https://mlsanity.com/paper/2410.18038/) (2024)
- [LLM Inference Serving: Survey of Recent Advances and Opportunities](https://mlsanity.com/paper/2407.12391/) (2024)
- [Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services](https://mlsanity.com/paper/2404.16283/) (2024)
- [NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference](https://mlsanity.com/paper/2411.01142/) (2024)
- [OmniRouter: Budget and Performance Controllable Multi-LLM Routing](https://mlsanity.com/paper/2502.20576/) (2025)
- [InferCept: Efficient Intercept Support for Augmented Large Language Model Inference](https://mlsanity.com/paper/2402.01869/) (2024)
- [Teola: Towards End-to-End Optimization of LLM-based Applications](https://mlsanity.com/paper/2407.00326/) (2024)
- [MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism](https://mlsanity.com/paper/2504.02263/) (2025)
- [M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity](https://mlsanity.com/paper/2404.14527/) (2024)
- [A Survey of Small Language Models](https://mlsanity.com/paper/2410.20011/) (2024)
- [P/D-Serve: Serving Disaggregated Large Language Model at Scale](https://mlsanity.com/paper/2408.08147/) (2024)
- [Unified Parameter-Efficient Unlearning for LLMs](https://mlsanity.com/paper/2412.00383/) (2024)
- [A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models](https://mlsanity.com/paper/2410.07265/) (2024)
- [ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments](https://mlsanity.com/paper/2502.09334/) (2025)
- [Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs](https://mlsanity.com/paper/2502.00722/) (2025)
- [SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving](https://mlsanity.com/paper/2408.05235/) (2024)
- [Online Scheduling for LLM Inference with KV Cache Constraints](https://mlsanity.com/paper/2502.07115/) (2025)
- [SLOs-Serve: Optimized Serving of Multi-SLO LLMs](https://mlsanity.com/paper/2504.08784/) (2025)
- [JITServe: SLO-aware LLM Serving with Imprecise Request Information](https://mlsanity.com/paper/2504.20068/) (2025)
- [Efficiently Serving Large Multimodal Models Using EPD Disaggregation](https://mlsanity.com/paper/2501.05460/) (2024)

## Abstract (arXiv)

Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase. However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference. To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in TetriInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computationsaturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots. Results show that TetriInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in turns of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively.