mlsanityData to 7 Oct 2026

Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads

arXiv 2401.11181 ↗First posted 20 Jan 2024cs.DCcited by 120

Not on the map. Why?

Cited by 77 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase. However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference. To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in TetriInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computationsaturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots. Results show that TetriInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in turns of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Jan 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 36; 2025: 49; 2026: 35.

120 papers.

Sort:

By area:

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  2. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  3. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  4. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  5. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  6. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  7. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  8. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  9. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  10. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  11. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  12. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 45
38 more
  1. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 44
  2. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  3. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  4. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inferencecited by 32
  5. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  6. 2024A Survey of Small Language Modelscited by 31
  7. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 31
  8. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  9. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  10. 2024Unified Parameter-Efficient Unlearning for LLMscited by 24
  11. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environmentscited by 23
  12. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  13. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 21
  14. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18
  15. 2025Online Scheduling for LLM Inference with KV Cache Constraintscited by 17
  16. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 17
  17. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
  18. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16
  19. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  20. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  21. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  22. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storagecited by 15
  23. 2025PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inferencecited by 14
  24. 2026LLM-42: Enabling Determinism in LLM Inference with Verified Speculationcited by 14
  25. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Servingcited by 14
  26. 2024vTensor: Flexible Virtual Tensor Management for Efficient LLM Servingcited by 13
  27. 2025IC-Cache: Efficient Large Language Model Serving via In-context Cachingcited by 13
  28. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Servingcited by 12
  29. 2024Aladdin: Joint Placement and Scaling for SLO-Aware LLM Servingcited by 11
  30. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scalecited by 11
  31. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Servingcited by 10
  32. 2024GreenLLM: Disaggregating Large Language Model Serving on Heterogeneous GPUs for Lower Carbon Emissionscited by 10
  33. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  34. 2025HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inferencecited by 10
  35. 2024Etalon: Holistic Performance Evaluation Framework for LLM Inference Systemscited by 9
  36. 2025Beyond the Buzz: A Pragmatic Take on Inference Disaggregationcited by 9
  37. 2025FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusterscited by 9
  38. 2026Predicting LLM Output Length via Entropy-Guided Representationscited by 9

50 most cited of 120 All 120 (JSON)