mlsanityData to 7 Oct 2026

Boosting Long-Context Management via Query-Guided Activation Refilling

arXiv 2412.12486 ↗First posted 17 Dec 2024cs.CL cs.AI cs.IRcited by 1

Not on the map. Why?

Abstract (arXiv)

Processing long contexts poses a significant challenge for large language models (LLMs) due to their inherent context-window limitations and the computational burden of extensive key-value (KV) activations, which severely impact efficiency. For information-seeking tasks, full context perception is often unnecessary, as a query's information needs can dynamically range from localized details to a global perspective, depending on its complexity. However, existing methods struggle to adapt effectively to these dynamic information needs. In the paper, we propose a method for processing long-context information-seeking tasks via query-guided Activation Refilling (ACRE). ACRE constructs a Bi-layer KV Cache for long contexts, where the layer-1 (L1) cache compactly captures global information, and the layer-2 (L2) cache provides detailed and localized information. ACRE establishes a proxying relationship between the two caches, allowing the input query to attend to the L1 cache and dynamically refill it with relevant entries from the L2 cache. This mechanism integrates global understanding with query-specific local details, thus improving answer decoding. Experiments on a variety of long-context information-seeking datasets demonstrate ACRE's effectiveness, achieving improvements in both performance and efficiency.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

29 of 42 references.

References · 29

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  8. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  10. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  12. 2017The NarrativeQA Reading Comprehension Challengecited by 590
17 more
  1. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  2. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  3. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  4. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  5. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  6. 2024RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generationcited by 144
  7. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 140
  8. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  9. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 118
  10. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  11. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  12. 2024CompAct: Compressing Retrieved Documents Actively for Question Answeringcited by 64
  13. 2023A Survey on Long Text Modeling with Transformerscited by 47
  14. 2024Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMscited by 30
  15. 2024Are Long-LLMs A Necessity For Long-Context Tasks?cited by 22
  16. 2024Extending Llama-3's Context Ten-Fold Overnightcited by 15
  17. 2024Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimizationcited by 9

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120

Similar, not cited

Close in wording, no citation either way

  1. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
  2. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  3. 2025CacheFocus: Dynamic Cache Re-Positioning for Efficient Retrieval-Augmented Generation
  4. 2025Retrospective Sparse Attention for Efficient Long-Context Generation
  5. 2026Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context Language Models
  6. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods
  7. 2025CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation
  8. 2026OVAL: Output-Aware Local Page Bases for KV Cache Retrieval