# ContextPilot: Fast Long-Context Inference via Context Reuse

arXiv: https://arxiv.org/abs/2511.03475 · HTML: https://mlsanity.com/paper/2511.03475/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Efficient reasoning and agent inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 54 of its 73 references that are in the map.
- First posted on arXiv 5 Nov 2025.

## Placed in

- Efficient LLM inference and serving / [Efficient reasoning and agent inference](https://mlsanity.com/inference/efficient-reasoning-and-agent-inference/)

## Builds on (most cited first, 30 of 54)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models](https://mlsanity.com/paper/2510.04618/) (2025)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries](https://mlsanity.com/paper/2401.15391/) (2024)
- [MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent](https://mlsanity.com/paper/2507.02259/) (2025)
- [CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://mlsanity.com/paper/2405.16444/) (2024)
- [Chain of Agents: Large Language Models Collaborating on Long-Context Tasks](https://mlsanity.com/paper/2406.02818/) (2024)
- [ACON: Optimizing Context Compression for Long-horizon LLM Agents](https://mlsanity.com/paper/2510.00615/) (2025)
- [Context Length Alone Hurts LLM Performance Despite Perfect Retrieval](https://mlsanity.com/paper/2510.05381/) (2025)
- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference](https://mlsanity.com/paper/2510.09665/) (2025)
- [DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning](https://mlsanity.com/paper/2402.17453/) (2024)
- [Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG](https://mlsanity.com/paper/2410.05983/) (2024)
- [Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach](https://mlsanity.com/paper/2407.16833/) (2024)
- [General-Purpose In-Context Learning by Meta-Learning Transformers](https://mlsanity.com/paper/2212.04458/) (2022)
- [RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.12457/) (2024)
- [PQCache: Product Quantization-based KVCache for Long Context LLM Inference](https://mlsanity.com/paper/2407.12820/) (2024)
- [Reducing hallucination in structured outputs via Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.08189/) (2024)
- [Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems](https://mlsanity.com/paper/2407.01370/) (2024)
- [ServerlessLLM: Low-Latency Serverless Inference for Large Language Models](https://mlsanity.com/paper/2401.14351/) (2024)
- [KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse](https://mlsanity.com/paper/2502.16002/) (2025)

## Built on by (most cited first, 8 of 8)

- [MEPIC: Memory Efficient Position Independent Caching for LLM Serving](https://mlsanity.com/paper/2512.16822/) (2025)
- [CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration](https://mlsanity.com/paper/2604.25080/) (2026)
- [PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design](https://mlsanity.com/paper/2605.08581/) (2026)
- [Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics](https://mlsanity.com/paper/2606.01502/) (2026)
- [CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inference](https://mlsanity.com/paper/2606.19667/) (2026)
- [KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints](https://mlsanity.com/paper/2609.10266/) (2026)
- [FinCacheServe: Dependency-Consistent Answer Reuse for Cost-Efficient RAG Serving over Mutable Enterprise Documents](https://mlsanity.com/paper/2607.26076/) (2026)
- [Evaluating the accuracy of KV cache reuse techniques](https://mlsanity.com/paper/2609.31415/) (2026)

## Abstract (arXiv)

AI applications increasingly depend on long-context inference, where LLMs consume substantial context to support stronger reasoning. Common examples include retrieval-augmented generation, agent memory layers, and multi-agent orchestration. As input contexts get longer, prefill latency becomes the main bottleneck. Yet today's prefill acceleration techniques face a trade-off: they either preserve reasoning quality but deliver little KV-cache reuse, or improve reuse at the cost of degraded reasoning quality. We present ContextPilot, a system that accelerates prefill by introducing context reuse as a new mechanism for faster long-context inference. ContextPilot introduces a context index to identify overlapping context blocks across LLM interactions (e.g., across users and turns). It further proposes context ordering and de-duplication techniques to maximize KV-cache reuse. To preserve reasoning quality under reuse, it introduces succinct context annotations that prevent quality degradation. Finally, ContextPilot is built around a modular architecture with a clean interface that integrates with existing inference engines. Extensive evaluation shows that ContextPilot reduces LLM prefill latency by up to 3× compared to state-of-the-art methods while preserving reasoning quality. At longer context lengths, it can even improve reasoning quality. ContextPilot is open-sourced at: https://github.com/EfficientContext/ContextPilot.