Finch: Prompt-guided Key-Value Cache Compression
On the map: Retrieval-augmented generation → Generation and context integration
also in 1 more
Abstract (arXiv)
Recent large language model applications, such as Retrieval-Augmented Generation and chatbots, have led to an increased need to process longer input contexts. However, this requirement is hampered by inherent limitations. Architecturally, models are constrained by a context window defined during training. Additionally, processing extensive texts requires substantial GPU memory. We propose a novel approach, Finch, to compress the input context by leveraging the pre-trained model weights of the self-attention. Given a prompt and a long text, Finch iteratively identifies the most relevant Key (K) and Value (V) pairs over chunks of the text conditioned on the prompt. Only such pairs are stored in the KV cache, which, within the space constrained by the context window, ultimately contains a compressed version of the long text. Our proposal enables models to consume large inputs even with high compression (up to 93x) while preserving semantic integrity without the need for fine-tuning.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 42 of its 53 references that are in the map. First posted on arXiv 31 Jul 2024.
Builds on
42 of 53 references.
Closest ancestors · 1
Same direction, newest first
Other references · 41
- 2017Attention Is All You Needcited by 59,932
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
- 2023Mistral 7Bcited by 3,065
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
29 more
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
- 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
- 2022A Survey on In-context Learningcited by 717
- 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
- 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
- 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
- 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
- 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
- 2023Learning to Compress Prompts with Gist Tokenscited by 379
- 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
- 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
- 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
- 2023Language Modeling Is Compressioncited by 225
- 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
- 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
- 2021Learned Token Pruning for Transformerscited by 138
- 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
- 2024Transformers are Multi-State RNNscited by 120
- 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 96
- 2017Learning to Remember Rare Eventscited by 63
- 2023Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filteringcited by 62
- 2017Generating High-Quality and Informative Conversation Responses with Sequence-to-Sequence Modelscited by 42
- 2024On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inferencecited by 32
- 2022Transkimmer: Transformer Learns to Layer-wise Skimcited by 21
- 2022AdapLeR: Speeding up Inference by Adaptive Length Reductioncited by 19
- 2022Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selectioncited by 13
- 2024Memory GAPS: Would LLMs pass the Tulving Test?cited by 1
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
- 2026Learning What to Remember: Test-Time Training via Context Distillation3 Aug 2026
- 2025OSCAR: Online Soft Compression And Reranking17 Mar 2025
Similar, not cited
Close in wording, no citation either way
- 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
- 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
- 2025CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation
- 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning
- 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
- 2025FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension
- 2025Task-agnostic Prompt Compression with Context-aware Sentence Embedding and Reward-guided Task Descriptor
- 2026Behavior-Preserving KV Cache Compression