mlsanityData to 7 Oct 2026

APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding

arXiv 2502.05431 ↗First posted 8 Feb 2025cs.LG cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

Context-augmented generation (CAG) techniques, including RAG and ICL, require the efficient combination of multiple contexts to generate responses to user queries. Directly inputting these contexts as a sequence introduces a considerable computational burden by re-encoding the combined selection of contexts for every request. To address this, we explore the promising potential of parallel encoding to independently pre-compute and cache each context's KV states. This approach enables the direct loading of cached states during inference while accommodating more contexts through position reuse across contexts. However, due to misalignments in attention distribution, directly applying parallel encoding results in a significant performance drop. To enable effective and efficient CAG, we propose Adaptive Parallel Encoding (APE), which brings shared prefix, attention temperature, and scaling factor to align the distribution of parallel encoding with sequential encoding. Results on RAG and ICL tasks demonstrate that APE can preserve 98% and 93% sequential encoding performance using the same inputs while outperforming parallel encoding by 3.6% and 7.9%, respectively. It also scales to many-shot CAG, effectively encoding hundreds of contexts in parallel. Efficiency evaluation shows that APE can achieve an end-to-end 4.5× speedup by reducing 28× prefilling time for a 128K-length context.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Feb 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 15.

29 papers.

Sort:

By area:

  1. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 59
  2. 2025Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generationcited by 30
  3. 2025CacheClip: Accelerating RAG with Effective KV Cache Reusecited by 14
  4. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolationcited by 13
  5. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  6. 2025A3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Servingcited by 7
  7. 2025Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correctioncited by 6
  8. 2025Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Modelscited by 6
  9. 2026RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reusecited by 6
  10. 2026An experimental study of KV cache reuse strategies in chunk-level caching systemscited by 5
  11. 2025Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluationcited by 4
  12. 2026KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasingcited by 4
17 more
  1. 2025APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUscited by 3
  2. 2025Weak-to-Strong GraphRAG: Aligning Weak Retrievers with Large Language Models for Graph-based Retrieval Augmented Generationcited by 3
  3. 2026Parallel Context-of-Experts Decoding for Retrieval Augmented Generationcited by 1
  4. 2026CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residualscited by 1
  5. 2026What It Costs to Compose, Rebuild, and Correct Precomputed Memorycited by 1
  6. 2025PEVLM: Parallel Encoding for Vision-Language Modelscited by 0
  7. 2025Struc-EMB: The Potential of Structure-Aware Encoding in Language Embeddingscited by 0
  8. 2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placementcited by 0
  9. 2026Context Memorization for Efficient Long Context Generationcited by 0
  10. 2026Towards Direct Latent-Space Synthesis for Parallel Branches in LLM-Agent Workflowscited by 0
  11. 2026MaSRead: Content-Addressed Reading of Replicated Latent Storescited by 0
  12. 2026PARSER: Read in Parallel, Reason in Depth for Long-Context LLM Agentscited by 0
  13. 2026Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?cited by 0
  14. 2026KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpointscited by 0
  15. 2026Evaluating the accuracy of KV cache reuse techniquescited by 0
  16. 2026CacheRepair: Learning to Repair Cross-Chunk Context in RAG for KV Cache Fusioncited by 0
  17. 2026ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SPARC-RAG: Adaptive Sequential-Parallel Scaling with Context Management for Retrieval-Augmented Generation
  2. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding
  3. 2025Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decoding
  4. 2025Enhancing Cache-Augmented Generation (CAG) with Adaptive Contextual Compression for Scalable Knowledge Integration
  5. 2024Inference Scaling for Long-Context Retrieval Augmented Generation
  6. 2025Retrospective Sparse Attention for Efficient Long-Context Generation
  7. 2024Context Embeddings for Efficient Answer Generation in RAG
  8. 2025Enhancing RAG Efficiency with Adaptive Context Compression