mlsanityData to 7 Oct 2026

MIST: A Co-Design Framework for Heterogeneous, Multi-Stage LLM Inference

arXiv 2504.09775 ↗First posted 14 Apr 2025cs.AR cs.AI cs.DCcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Modern LLM serving now spans multi-stage pipelines including RAG retrieval and KV cache reuse, each with distinct compute, memory, and latency demands. Inference engines expose a large configuration space with no systematic navigation methodology, and exhaustively benchmarking configurations can exceed 40K in cloud costs. Simultaneously, the hardware landscape is rapidly diversifying across AMD GPUs, TPUs, and custom ASICs, while cross-vendor prefill-decode (PD) disaggregated configurations lack unified software stacks for end-to-end evaluation today. To address this gap, we present MIST, a Heterogeneous Multi-stage LLM inference Execution Simulator. MIST models diverse request stages; including RAG, KV retrieval, reasoning, prefill, and decode across complex hardware hierarchies. MIST supports heterogeneous clients executing multiple models concurrently unlike prior frameworks while incorporating advanced batching strategies and multi-level memory hierarchies. By integrating real hardware traces with analytical modeling, MIST captures critical trade-offs such as memory bandwidth contention, inter-cluster communication latency, and batching efficiency in hybrid CPU-accelerator deployments. Through case studies, we explore the impact of reasoning stages on end-to-end latency, optimal batching strategies for hybrid pipelines, and the architectural implications of remote KV cache retrieval. MIST empowers system designers to navigate the evolving landscape of LLM inference, providing actionable insights into optimizing hardware-software co-design for next-generation AI workloads.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 55 references that are in the map. First posted on arXiv 14 Apr 2025.

Builds on

25 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale10 Aug
  2. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models3 Jun
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 21

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  7. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  10. 2024A Survey on Large Language Models for Code Generationcited by 646
  11. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
9 more
  1. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  2. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  3. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  4. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  5. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  6. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  7. 2024Do Large Language Models Need a Content Delivery Network?cited by 15
  8. 2023A Hardware Evaluation Framework for Large Language Model Inferencecited by 14
  9. 2024Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  2. 2026How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Servingcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
  2. 2026Frontier: Towards Comprehensive and Accurate LLM Inference Simulation
  3. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  4. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
  5. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  6. 2026SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybridity
  7. 2026LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
  8. 2026FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference