mlsanityData to 7 Oct 2026

ALR2: A Retrieve-then-Reason Framework for Long-context Question Answering

arXiv 2410.03227 ↗First posted 4 Oct 2024cs.CLcited by 25

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The context window of large language models (LLMs) has been extended significantly in recent years. However, while the context length that the LLM can process has grown, the capability of the model to accurately reason over that context degrades noticeably. This occurs because modern LLMs often become overwhelmed by the vast amount of information in the context; when answering questions, the model must identify and reason over relevant evidence sparsely distributed throughout the text. To alleviate the challenge of long-context reasoning, we develop a retrieve-then-reason framework, enabling LLMs to reason over relevant evidence collected during an intermediate retrieval step. We find that modern LLMs struggle to accurately retrieve relevant facts and instead, often hallucinate "retrieved facts", resulting in flawed reasoning and the production of incorrect answers. To address these issues, we introduce ALR2, a method that augments the long-context reasoning capability of LLMs via an explicit two-stage procedure, i.e., aligning LLMs with the objectives of both retrieval and reasoning. We demonstrate the efficacy of ALR2 for mitigating performance degradation in long-context reasoning tasks. Through extensive experiments on long-context QA benchmarks, we find our method to outperform competitive baselines by large margins, achieving at least 8.4 and 7.9 EM gains on the long-context versions of HotpotQA and SQuAD datasets, respectively.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 56 of its 62 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

56 of 62 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack14 Jun
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Models6 Oct

Other references · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  12. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
38 more
  1. 2020Longformer: The Long-Document Transformercited by 3,362
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  10. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  11. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  12. 2022Galactica: A Large Language Model for Sciencecited by 736
  13. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  14. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  15. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  16. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  17. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  18. 2023Adapting Language Models to Compress Contextscited by 350
  19. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  20. 2022Teaching language models to support answers with verified quotescited by 290
  21. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  22. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  23. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  24. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  25. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  26. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
  27. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  28. 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 112
  29. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 110
  30. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  31. 2023System 2 Attention (is something you might need too)cited by 85
  32. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  33. 2016Improving Neural Language Models with a Continuous Cachecited by 81
  34. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  35. 2021Neural Machine Translation with Monolingual Translation Memorycited by 61
  36. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 55
  37. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 53
  38. 2024Base of RoPE Bounds Context Lengthcited by 49

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 17; 2026: 5.

25 papers.

Sort:

By area:

  1. 2025In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agentscited by 102
  2. 2024o1-Coder: an o1 Replication for Codingcited by 77
  3. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 48
  4. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 40
  5. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  6. 2025LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contextscited by 25
  7. 2025SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environmentscited by 18
  8. 2024OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuningcited by 13
  9. 2025Eliciting In-context Retrieval and Reasoning for Long-context Large Language Modelscited by 11
  10. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  11. 2025Positional Biases Shift as Inputs Approach Context Window Limitscited by 9
  12. 2025Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrievalcited by 5
13 more
  1. 2025Improving Context Fidelity via Native Retrieval-Augmented Reasoningcited by 5
  2. 2025Revisiting Long-context Modeling from Context Denoising Perspectivecited by 5
  3. 2025OpenRAG: Optimizing RAG End-to-End via In-Context Retrieval Learningcited by 4
  4. 2025CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capabilitycited by 4
  5. 2025LongReasonArena: A Long Reasoning Benchmark for Large Language Modelscited by 3
  6. 2025COSMIR: Chain Orchestrated Structured Memory for Iterative Reasoning over Long Contextcited by 1
  7. 2025Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generationcited by 1
  8. 2025StorageXTuner: An LLM Agent-Driven Automatic Tuning Framework for Heterogeneous Storage Systemscited by 1
  9. 2026Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shapingcited by 1
  10. 2026LEAD: Breaking the No-Recovery Bottleneck in Long-Horizon Reasoningcited by 1
  11. 2026A Decomposition Perspective to Long-context Reasoning for LLMscited by 1
  12. 2026RecaLLM: Addressing the Lost-in-Thought Phenomenon with Explicit In-Context Retrievalcited by 1
  13. 2026What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering
  2. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion
  3. 2026π2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models
  4. 2026ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning
  5. 2024Long-context Language Models Fail in Basic Retrieval Tasks Without Sufficient Reasoning Steps
  6. 2023Marathon: A Race Through the Realm of Long Context with Large Language Models
  7. 2026Context-Length Robustness in Question Answering Models: A Comparative Empirical Study
  8. 2025Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-k