mlsanityData to 7 Oct 2026

CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capability

arXiv 2505.10063 ↗First posted 15 May 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Advancements in Large Language Models (LLMs) have extended their input context length, yet they still struggle with retrieval and reasoning in long-context inputs. Existing methods propose to utilize the prompt strategy and retrieval head to alleviate this limitation. However, they still face challenges in balancing retrieval precision and recall, impacting their efficacy in answering questions. To address this, we introduce CAFE, a two-stage coarse-to-fine method to enhance multi-document question-answering capacities. By gradually eliminating the negative impacts of background and distracting documents, CAFE makes the responses more reliant on the evidence documents. Initially, a coarse-grained filtering method leverages retrieval heads to identify and rank relevant documents. Then, a fine-grained steering method guides attention to the most relevant content. Experiments across benchmarks show CAFE outperforms baselines, achieving up to 22.1% and 13.7% SubEM improvement over SFT and RAG methods on the Mistral model, respectively.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 15 May 2025.

Builds on

32 of 41 references.

References · 32

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  10. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  11. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  12. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
20 more
  1. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  2. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  3. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  4. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  5. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
  6. 2024Make Your LLM Fully Utilize the Contextcited by 118
  7. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 99
  8. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  9. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82
  10. 2024RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinementcited by 38
  11. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  12. 2024Making Long-Context Language Models Better Multi-Hop Reasonerscited by 25
  13. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answeringcited by 25
  14. 2024DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinationscited by 23
  15. 2024How do Large Language Models Learn In-Context? Query and Key Matrices of In-Context Heads are Two Towers for Metric Learningcited by 18
  16. 2024HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolationcited by 17
  17. 2024Model Tells Itself Where to Attend: Faithfulness Meets Automatic Attention Steeringcited by 15
  18. 2025LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillationcited by 14
  19. 2025Eliciting In-context Retrieval and Reasoning for Long-context Large Language Modelscited by 11
  20. 2024Can't Remember Details in Long Documents? You Need Some R&Rcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 21
  2. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 8
  3. 2025Bridging External and Parametric Knowledge: Mitigating Hallucination of LLMs with Shared-Private Semantic Synergy in Dual-Stream Knowledgecited by 3
  4. 2026Beyond Memory Majority: Latent-Source Reasoning for Multi-Agent Memory Arbitrationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Leveraging Inter-Chunk Interactions for Enhanced Retrieval in Large Language Model-Based Question Answering
  2. 2025PluriHopRAG: Exhaustive, Recall-Sensitive QA Through Corpus-Specific Document Structure Learning
  3. 2026ConRAG: Consensus-Driven Multi-View Retrieval for Multi-Hop Question Answering
  4. 2024Hierarchical Retrieval-Augmented Generation Model with Rethink for Multi-hop Question Answering
  5. 2026Qwen Goes Brrr: Off-the-Shelf RAG for Ukrainian Multi-Domain Document Understanding
  6. 2026PAR2-RAG: Planned Active Retrieval and Reasoning for Multi-Hop Question Answering
  7. 2026MARA: A Multimodal Adaptive Retrieval-Augmented Framework for Document Question Answering
  8. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions