# CacheFocus: Dynamic Cache Re-Positioning for Efficient Retrieval-Augmented Generation

arXiv: https://arxiv.org/abs/2502.11101 · HTML: https://mlsanity.com/paper/2502.11101/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 16 Feb 2025.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)

## Builds on (most cited first, 19 of 19)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens](https://mlsanity.com/paper/2402.13753/) (2024)
- [PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training](https://mlsanity.com/paper/2309.10400/) (2023)
- [Structured Prompting: Scaling In-Context Learning to 1,000 Examples](https://mlsanity.com/paper/2212.06713/) (2022)
- [Parallel Context Windows for Large Language Models](https://mlsanity.com/paper/2212.10947/) (2022)
- [CLEX: Continuous Length Extrapolation for Large Language Models](https://mlsanity.com/paper/2310.16450/) (2023)
- [Block-Attention for Efficient Prefilling](https://mlsanity.com/paper/2409.15355/) (2024)
- [Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection](https://mlsanity.com/paper/2405.16178/) (2024)
- [LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models](https://mlsanity.com/paper/2409.00509/) (2024)
- [Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.06910/) (2024)
- [FocusLLM: Precise Understanding of Long Context by Dynamic Condensing](https://mlsanity.com/paper/2408.11745/) (2024)
- [XL3M: A Training-free Framework for LLM Length Extension Based on Segment-wise Inference](https://mlsanity.com/paper/2405.17755/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) excel across a variety of language tasks yet are constrained by limited input lengths and high computational costs. Existing approachestextemdash such as relative positional encodings (e.g., RoPE, ALiBi) and sliding window mechanismstextemdash partially alleviate these issues but often require additional training or suffer from performance degradation with longer inputs. In this paper, we introduce textbftextitCacheFocus, a method that enhances length normalization and reduces inference latency without any further training. Our approach leverages query-independent, offline caching to efficiently reuse a Context KV Cache Store. We address the amplification of abnormal token distributions problem by re-positioning cached keys and introducing Layer-Adaptive Cache Pruning to discard low-relevance caches during pre-filling. Additionally, our Adaptive Positional Allocation Strategy dynamically reassigns cache positions to maximize the use of the available positional encoding range. Experiments on the Natural Questions and TriviaQA datasets demonstrate that CacheFocus outperforms alternative methods even when inputs exceed the 4K limit of the textttLLaMA-2 model, emphasizing its practical effectiveness for long-context LLMs. Moreover, even with large maximum input length of textttQwen2, the performance of CacheFocus shows that it maintains consistent performance even as the number of documents increases, effectively managing long-text generation without degradation.