mlsanityData to 7 Oct 2026

Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?

arXiv 2609.09768 ↗First posted 9 Sep 2026cs.LG cs.AI cs.CLcited by 0

On the map: Retrieval-augmented generation → Generation and context integration

also in 1 more
Abstract (arXiv)

In Retrieval-Augmented Generation (RAG) systems, a large number of retrieved chunks are concatenated to form the input context so that users can receive high-quality responses based on external knowledge. As a result, the input context length increases substantially, leading to a larger prefill workload and, in turn, a longer time to first token (TTFT). While previous works that reuse precomputed key-value (KV) caches effectively reduce TTFT for long-context inputs, it remains unclear whether response quality is preserved when the input context becomes very long. In this paper, we propose a combined approach that (i) fine-tunes the model while taking KV cache concatenation into account and (ii) selectively recomputes a subset of the KV caches. By applying both techniques, we demonstrate improved accuracy for long-context inputs. Experiments on the RULER benchmark show that, for a 124k-token input, our method improves the RULER score by 9.7 point over the baseline that recomputes KV caches only. Moreover, TTFT is reduced by 80% compared with full attention.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 31 of its 39 references that are in the map. First posted on arXiv 9 Sep 2026.

Builds on

31 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026An experimental study of KV cache reuse strategies in chunk-level caching systems3 Mar
  2. 2025A3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Serving13 Nov
  3. 2025CacheClip: Accelerating RAG with Effective KV Cache Reuse11 Oct
  4. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text10 Oct

Other references · 27

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
15 more
  1. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  2. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  5. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  6. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  7. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  8. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  9. 2022A Survey on Retrieval-Augmented Text Generationcited by 157
  10. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  11. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  12. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 45
  13. 2025APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encodingcited by 29
  14. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reusecited by 22
  15. 2024Let the Code LLM Edit Itself When You Edit the Codecited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation
  2. 2026ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation
  3. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation
  4. 2026PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Serving
  5. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
  6. 2026QCFuse: Query-Aware Cache Fusion via Compressed View for Efficient RAG Serving
  7. 2026KV2: A Self-Refining KV Cache
  8. 2025HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse