# APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding

arXiv: https://arxiv.org/abs/2502.05431 · HTML: https://mlsanity.com/paper/2502.05431/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2025.

## Built on by (most cited first, 29 of 29)

- [Parallel-R1: Towards Parallel Thinking via Reinforcement Learning](https://mlsanity.com/paper/2509.07980/) (2025)
- [Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generation](https://mlsanity.com/paper/2506.09991/) (2025)
- [CacheClip: Accelerating RAG with Effective KV Cache Reuse](https://mlsanity.com/paper/2510.10129/) (2025)
- [ParallelComp: Parallel Long-Context Compressor for Length Extrapolation](https://mlsanity.com/paper/2502.14317/) (2025)
- [ContextPilot: Fast Long-Context Inference via Context Reuse](https://mlsanity.com/paper/2511.03475/) (2025)
- [A3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Serving](https://mlsanity.com/paper/2511.17560/) (2025)
- [RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse](https://mlsanity.com/paper/2603.13289/) (2026)
- [Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction](https://mlsanity.com/paper/2505.11254/) (2025)
- [Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Models](https://mlsanity.com/paper/2506.07334/) (2025)
- [KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasing](https://mlsanity.com/paper/2606.17034/) (2026)
- [An experimental study of KV cache reuse strategies in chunk-level caching systems](https://mlsanity.com/paper/2603.20218/) (2026)
- [Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation](https://mlsanity.com/paper/2510.07414/) (2025)
- [APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUs](https://mlsanity.com/paper/2502.12085/) (2025)
- [Weak-to-Strong GraphRAG: Aligning Weak Retrievers with Large Language Models for Graph-based Retrieval Augmented Generation](https://mlsanity.com/paper/2506.22518/) (2025)
- [PEVLM: Parallel Encoding for Vision-Language Models](https://mlsanity.com/paper/2506.19651/) (2025)
- [Parallel Context-of-Experts Decoding for Retrieval Augmented Generation](https://mlsanity.com/paper/2601.08670/) (2026)
- [CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residuals](https://mlsanity.com/paper/2607.28263/) (2026)
- [What It Costs to Compose, Rebuild, and Correct Precomputed Memory](https://mlsanity.com/paper/2608.30647/) (2026)
- [KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints](https://mlsanity.com/paper/2609.10266/) (2026)
- [Struc-EMB: The Potential of Structure-Aware Encoding in Language Embeddings](https://mlsanity.com/paper/2510.08774/) (2025)
- [HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placement](https://mlsanity.com/paper/2510.20878/) (2025)
- [Context Memorization for Efficient Long Context Generation](https://mlsanity.com/paper/2605.18226/) (2026)
- [Towards Direct Latent-Space Synthesis for Parallel Branches in LLM-Agent Workflows](https://mlsanity.com/paper/2606.14672/) (2026)
- [MaSRead: Content-Addressed Reading of Replicated Latent Stores](https://mlsanity.com/paper/2608.11218/) (2026)
- [PARSER: Read in Parallel, Reason in Depth for Long-Context LLM Agents](https://mlsanity.com/paper/2609.06702/) (2026)
- [Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?](https://mlsanity.com/paper/2609.09768/) (2026)
- [Evaluating the accuracy of KV cache reuse techniques](https://mlsanity.com/paper/2609.31415/) (2026)
- [CacheRepair: Learning to Repair Cross-Chunk Context in RAG for KV Cache Fusion](https://mlsanity.com/paper/2609.35139/) (2026)
- [ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptation](https://mlsanity.com/paper/2609.36722/) (2026)

## Abstract (arXiv)

Context-augmented generation (CAG) techniques, including RAG and ICL, require the efficient combination of multiple contexts to generate responses to user queries. Directly inputting these contexts as a sequence introduces a considerable computational burden by re-encoding the combined selection of contexts for every request. To address this, we explore the promising potential of parallel encoding to independently pre-compute and cache each context's KV states. This approach enables the direct loading of cached states during inference while accommodating more contexts through position reuse across contexts. However, due to misalignments in attention distribution, directly applying parallel encoding results in a significant performance drop. To enable effective and efficient CAG, we propose Adaptive Parallel Encoding (APE), which brings shared prefix, attention temperature, and scaling factor to align the distribution of parallel encoding with sequential encoding. Results on RAG and ICL tasks demonstrate that APE can preserve 98% and 93% sequential encoding performance using the same inputs while outperforming parallel encoding by 3.6% and 7.9%, respectively. It also scales to many-shot CAG, effectively encoding hundreds of contexts in parallel. Efficiency evaluation shows that APE can achieve an end-to-end 4.5× speedup by reducing 28× prefilling time for a 128K-length context.