# A3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Serving

arXiv: https://arxiv.org/abs/2511.17560 · HTML: https://mlsanity.com/paper/2511.17560/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 13 Nov 2025.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)
- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://mlsanity.com/paper/2501.01005/) (2025)
- [CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://mlsanity.com/paper/2405.16444/) (2024)
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM](https://mlsanity.com/paper/2403.05527/) (2024)
- [Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach](https://mlsanity.com/paper/2407.16833/) (2024)
- [LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism](https://mlsanity.com/paper/2404.09526/) (2024)
- [SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator](https://mlsanity.com/paper/2412.12094/) (2024)
- [APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding](https://mlsanity.com/paper/2502.05431/) (2025)
- [Let the Code LLM Edit Itself When You Edit the Code](https://mlsanity.com/paper/2407.03157/) (2024)

## Built on by (most cited first, 8 of 8)

- [MEPIC: Memory Efficient Position Independent Caching for LLM Serving](https://mlsanity.com/paper/2512.16822/) (2025)
- [KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs](https://mlsanity.com/paper/2604.13226/) (2026)
- [HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Caching](https://mlsanity.com/paper/2607.01299/) (2026)
- [SemPIC: Learning Semantic Position-Independent KV Caches](https://mlsanity.com/paper/2607.28069/) (2026)
- [Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?](https://mlsanity.com/paper/2609.09768/) (2026)
- [Evaluating the accuracy of KV cache reuse techniques](https://mlsanity.com/paper/2609.31415/) (2026)
- [ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptation](https://mlsanity.com/paper/2609.36722/) (2026)
- [Budgeted KV-Cache Repair for Evolving Contexts](https://mlsanity.com/paper/2610.17338/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated strong capabilities in processing long contexts, enabling them to tackle tasks involving long textual inputs such as multi-turn conversations, legal documents, or retrieved documents in Retrieval-Augmented Generation (RAG) systems. However, despite their ability to handle long sequences, the resulting decoding latency and memory overhead remain substantial, posing challenges for real-world deployment. Recent advances in KV Cache reuse have shown potential to mitigate these costs, but still suffer from notable performance degradation. To address this issue, we conduct an in-depth investigation of recomputation-based reuse methods and observe that the recomputed tokens often fail to align with the context segments most relevant to the question. This misalignment hinders proper updates to the critical contextual representations. Therefore, we propose the Attention-Aware Accurate KV Cache Fusion algorithm (A3), which precomputes and selectively fuses the KV Cache of text chunks based on their relevance to the question, achieving accurate integration with minimal computational overhead. Extensive experiments on various benchmarks and LLMs demonstrate that A3 achieves the best task performance compared to four baselines while reducing the time-to-first-token (TTFT) by 2×.