mlsanityData to 14 Oct 2026

PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design

arXiv 2605.08581 ↗First posted 9 May 2026cs.LGcited by 1

On the map: Retrieval-augmented generation → Generation and context integration

also in 1 more
Abstract (arXiv)

Online large language model (LLM) services process requests that share context but differ in prefix structure and arrival timing. Finite capacity couples request admission and key-value (KV) retention. This paper addresses one online serving problem: using a unified view of visible demand to choose the feasible joint admission and KV-retention candidate with the lowest predicted latency cost as prefix structures and arrival rates vary. To model this problem, we introduce HorizonSchema, a workload abstraction that connects exact prefix paths, permitted serializations, and released consumers as shared inputs to both decisions. Our analysis shows that useful reuse depends on demand and prefix survival, while service order changes latency even at equal computation. Furthermore, we present HAServe, which turns this abstraction into a scheduling--memory co-design method of systematic optimization for online LLM serving. We implement HAServe on SGLang and evaluate Qwen3-4B-Instruct-2507 and Llama2-13B on five workloads using eight H100 GPUs. Across the evaluated workloads, HAServe achieves up to 1.14--5.90× P99 TTFT speedups and up to 1.08--1.44× the throughput, with KV-cache hit rate up to 20.59 percentage points higher than the strongest baseline for each metric and setting.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 30 of its 33 references that are in the map. First posted on arXiv 9 May 2026.

Builds on

30 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text10 Oct
  2. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation18 Apr
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May

Other references · 27

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  8. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  9. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  10. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  11. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  12. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
15 more
  1. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  3. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  4. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 257
  5. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  6. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  7. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  8. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  9. 2024PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-designcited by 28
  10. 2024RAGGED: Towards Informed Design of Scalable and Stable RAG Systemscited by 24
  11. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Servingcited by 19
  12. 2025LLM Query Scheduling with Prefix Reuse and Latency Constraintscited by 9
  13. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  14. 2025RAGPulse: An Open-Source RAG Workload Trace to Optimize RAG Serving Systemscited by 2
  15. 2026Updatable Balanced Index for Stable Streaming Similarity Search over Large-Scale Fresh Vectorscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026CacheBridge: Efficient Cross-Model KV Cache Transfercited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM Serving Optimization with Variable Prefill and Decode Lengths
  2. 2026P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Serving
  3. 2024Fast Inference for Augmented Large Language Models
  4. 2025Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics
  5. 2026Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice
  6. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  7. 2025FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework
  8. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving