mlsanityData to 14 Oct 2026

Strata: Hierarchical Context Caching for Long Context Language Model Serving

arXiv 2508.18572 ↗First posted 26 Aug 2025cs.DCcited by 26

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Models (LLMs) with expanding context windows face significant performance hurdles. While caching key-value (KV) states is critical for avoiding redundant computation, the storage footprint of long-context caches quickly exceeds GPU memory capacity, forcing production systems to adopt hierarchical caching across memory hierarchies. However, transferring large cached contexts back to the GPU introduces severe performance bottlenecks: fragmented I/O from paged layouts prevents full bandwidth utilization, and existing schedulers fail to account for cache-loading delays, leaving systems loading-bound rather than compute-bound. We present Strata, a hierarchical context caching framework designed for efficient long context LLM serving. Strata introduces GPU-assisted I/O to combat KV cache fragmentation, decoupling GPU and CPU memory layouts and employs cache-aware request scheduling to balance compute with I/O latency and overlapping unavoidable stalls with complementary tasks. Built on SGLang and deployed in production, Strata achieves up to 5x lower Time-To-First-Token (TTFT) compared to vLLM + LMCache and 3.75x speedup over NVIDIA TensorRT-LLM on long-context benchmarks, without degrading short-context performance.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 40 references that are in the map. First posted on arXiv 26 Aug 2025.

Builds on

15 of 40 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Compute Or Load KV Cache? Why Not Both?4 Oct
  2. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  3. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  4. 2023Stateful Large Language Model Serving with Pensieve9 Dec
  5. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 9

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  2. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  3. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  4. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  5. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 90
  6. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
  7. 2024Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactionscited by 46
  8. 2022GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecturecited by 12
  9. 2020EMOGI: Efficient Memory-access for Out-of-memory Graph-traversal In GPUscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 21.

26 papers.

Sort:

By area:

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  2. 2026Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Servingcited by 20
  3. 2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference Systemcited by 17
  4. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 10
  5. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  6. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Servingcited by 6
  7. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inferencecited by 5
  8. 2026PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Servingcited by 4
  9. 2026PEEK: Context Map as an Orientation Cache for Long-Context LLM Agentscited by 4
  10. 2026Bole: Efficient Tree Speculation for Hybrid-Attention Language Modelscited by 3
  11. 2026HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Managementcited by 3
  12. 2025RepetitionCurse: Measuring and Understanding Router Imbalance in Mixture-of-Experts LLMs under DoS Stresscited by 2
14 more
  1. 2026ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cachecited by 2
  2. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inferencecited by 1
  3. 2026DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72cited by 1
  4. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restorationcited by 1
  5. 2026Diagnose Before You Compress: Prediction-Independent Bottleneck Witness Refinement for LLM Serving Tracescited by 1
  6. 2026vToken: Token-Level Virtualization for Reclaimable KV Cachescited by 1
  7. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Servingcited by 0
  8. 2026Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attentioncited by 0
  9. 2026Bounded-State Restoration: Decoupling Local Restore Capacity from External LLM Statecited by 0
  10. 2026SPLASH: Co-Designing Sparse Attention with High-Bandwidth Flash for Efficient Long-Context Inferencecited by 0
  11. 2026When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reusecited by 0
  12. 2026Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUscited by 0
  13. 2026TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flashcited by 0
  14. 2026Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management
  2. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management
  3. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  4. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
  5. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
  6. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving
  7. 2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving
  8. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption