mlsanityData to 7 Oct 2026

ContextPilot: Fast Long-Context Inference via Context Reuse

arXiv 2511.03475 ↗First posted 5 Nov 2025cs.LGcited by 8

On the map: Efficient LLM inference and serving → Efficient reasoning and agent inference

Abstract (arXiv)

AI applications increasingly depend on long-context inference, where LLMs consume substantial context to support stronger reasoning. Common examples include retrieval-augmented generation, agent memory layers, and multi-agent orchestration. As input contexts get longer, prefill latency becomes the main bottleneck. Yet today's prefill acceleration techniques face a trade-off: they either preserve reasoning quality but deliver little KV-cache reuse, or improve reuse at the cost of degraded reasoning quality. We present ContextPilot, a system that accelerates prefill by introducing context reuse as a new mechanism for faster long-context inference. ContextPilot introduces a context index to identify overlapping context blocks across LLM interactions (e.g., across users and turns). It further proposes context ordering and de-duplication techniques to maximize KV-cache reuse. To preserve reasoning quality under reuse, it introduces succinct context annotations that prevent quality degradation. Finally, ContextPilot is built around a modular architecture with a clean interface that integrates with existing inference engines. Extensive evaluation shows that ContextPilot reduces LLM prefill latency by up to 3× compared to state-of-the-art methods while preserving reasoning quality. At longer context lengths, it can even improve reasoning quality. ContextPilot is open-sourced at: https://github.com/EfficientContext/ContextPilot.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Efficient reasoning and agent inference, one of 12 directions in Efficient LLM inference and serving. Builds on 54 of its 73 references that are in the map. First posted on arXiv 5 Nov 2025.

Builds on

54 of 73 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025ACON: Optimizing Context Compression for Long-horizon LLM Agents1 Oct

Other references · 53

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  3. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  5. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  6. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
  7. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  8. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  9. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  10. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  11. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 349
  12. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
38 more
  1. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 253
  2. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  3. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  4. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  5. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  6. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  7. 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoningcited by 119
  8. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  9. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  10. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 117
  11. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  12. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  13. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  14. 2024Reducing hallucination in structured outputs via Retrieval-Augmented Generationcited by 92
  15. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  16. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  17. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  18. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 60
  19. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  20. 2025SagaLLM: Context Management, Validation, and Transaction Guarantees for Multi-Agent LLM Planningcited by 58
  21. 2025Hilbert: Recursively Building Formal Proofs with Informal Reasoningcited by 55
  22. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systemscited by 50
  23. 2025Contextual Experience Replay for Self-Improvement of Language Agentscited by 46
  24. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 38
  25. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36
  26. 2024Block-Attention for Efficient Prefillingcited by 34
  27. 2025Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQLcited by 30
  28. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  29. 2025APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encodingcited by 29
  30. 2025Strata: Hierarchical Context Caching for Long Context Language Model Servingcited by 25
  31. 2023Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Modelscited by 23
  32. 2024RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialoguescited by 19
  33. 2025Rethinking Chunk Size For Long-Document Retrieval: A Multi-Dataset Analysiscited by 17
  34. 2025Context Engineering 2.0: The Context of Context Engineeringcited by 17
  35. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decodingcited by 14
  36. 2025WaferLLM: Large Language Model Inference at Wafer Scalecited by 13
  37. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  38. 2023Zero-Shot Cross-Lingual Reranking with Large Language Models for Low-Resource Languagescited by 8

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 7.

8 papers.

  1. 2025MEPIC: Memory Efficient Position Independent Caching for LLM Servingcited by 10
  2. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restorationcited by 1
  3. 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-designcited by 1
  4. 2026Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabricscited by 1
  5. 2026CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inferencecited by 1
  6. 2026FinCacheServe: Dependency-Consistent Answer Reuse for Cost-Efficient RAG Serving over Mutable Enterprise Documentscited by 0
  7. 2026KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpointscited by 0
  8. 2026Evaluating the accuracy of KV cache reuse techniquescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ContextPilot: Teaching Agents for Proactive Context Management via Fine-grained RL
  2. 2026ContextWeaver: Selective and Dependency-Structured Memory Construction for LLM Agents
  3. 2026Akashic: A Low-Overhead LLM Inference Service with MemAttention
  4. 2026TokenPilot: Cache-Efficient Context Management for LLM Agents
  5. 2025PERK: Long-Context Reasoning as Test-Time Learning
  6. 2026PEEK: Context Map as an Orientation Cache for Long-Context LLM Agents
  7. 2026ContextPipe: Database-Inspired Context Assembly for Long-Horizon Agents
  8. 2026Beyond Memory: Harnessing Long-Horizon Agents with Explicit Belief States