# ALR2: A Retrieve-then-Reason Framework for Long-context Question Answering

arXiv: https://arxiv.org/abs/2410.03227 · HTML: https://mlsanity.com/paper/2410.03227/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 56 of its 62 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 56)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)

## Built on by (most cited first, 26 of 26)

- [In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents](https://mlsanity.com/paper/2503.08026/) (2025)
- [o1-Coder: an o1 Replication for Coding](https://mlsanity.com/paper/2412.00154/) (2024)
- [Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs](https://mlsanity.com/paper/2510.17771/) (2025)
- [Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs](https://mlsanity.com/paper/2507.09477/) (2025)
- [Large Language Models Can Self-Improve in Long-context Reasoning](https://mlsanity.com/paper/2411.08147/) (2024)
- [LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts](https://mlsanity.com/paper/2510.19363/) (2025)
- [SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments](https://mlsanity.com/paper/2507.09063/) (2025)
- [OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuning](https://mlsanity.com/paper/2412.16849/) (2024)
- [Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models](https://mlsanity.com/paper/2501.08248/) (2025)
- [WildLong: Synthesizing Realistic Long-Context Instruction Data at Scale](https://mlsanity.com/paper/2502.16684/) (2025)
- [Positional Biases Shift as Inputs Approach Context Window Limits](https://mlsanity.com/paper/2508.07479/) (2025)
- [Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrieval](https://mlsanity.com/paper/2503.09819/) (2025)
- [Improving Context Fidelity via Native Retrieval-Augmented Reasoning](https://mlsanity.com/paper/2509.13683/) (2025)
- [Revisiting Long-context Modeling from Context Denoising Perspective](https://mlsanity.com/paper/2510.05862/) (2025)
- [OpenRAG: Optimizing RAG End-to-End via In-Context Retrieval Learning](https://mlsanity.com/paper/2503.08398/) (2025)
- [CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capability](https://mlsanity.com/paper/2505.10063/) (2025)
- [LongReasonArena: A Long Reasoning Benchmark for Large Language Models](https://mlsanity.com/paper/2508.19363/) (2025)
- [Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shaping](https://mlsanity.com/paper/2601.12465/) (2026)
- [What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation](https://mlsanity.com/paper/2608.22948/) (2026)
- [COSMIR: Chain Orchestrated Structured Memory for Iterative Reasoning over Long Context](https://mlsanity.com/paper/2510.04568/) (2025)
- [Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.13191/) (2025)
- [StorageXTuner: An LLM Agent-Driven Automatic Tuning Framework for Heterogeneous Storage Systems](https://mlsanity.com/paper/2510.25017/) (2025)
- [LEAD: Breaking the No-Recovery Bottleneck in Long-Horizon Reasoning](https://mlsanity.com/paper/2603.06870/) (2026)
- [A Decomposition Perspective to Long-context Reasoning for LLMs](https://mlsanity.com/paper/2604.07981/) (2026)
- [RecaLLM: Addressing the Lost-in-Thought Phenomenon with Explicit In-Context Retrieval](https://mlsanity.com/paper/2604.09494/) (2026)
- [TrajLong: Co-Designing Agentic and Long-Context Supervision for Mid-Training](https://mlsanity.com/paper/2610.04973/) (2026)

## Abstract (arXiv)

The context window of large language models (LLMs) has been extended significantly in recent years. However, while the context length that the LLM can process has grown, the capability of the model to accurately reason over that context degrades noticeably. This occurs because modern LLMs often become overwhelmed by the vast amount of information in the context; when answering questions, the model must identify and reason over relevant evidence sparsely distributed throughout the text. To alleviate the challenge of long-context reasoning, we develop a retrieve-then-reason framework, enabling LLMs to reason over relevant evidence collected during an intermediate retrieval step. We find that modern LLMs struggle to accurately retrieve relevant facts and instead, often hallucinate "retrieved facts", resulting in flawed reasoning and the production of incorrect answers. To address these issues, we introduce ALR2, a method that augments the long-context reasoning capability of LLMs via an explicit two-stage procedure, i.e., aligning LLMs with the objectives of both retrieval and reasoning. We demonstrate the efficacy of ALR2 for mitigating performance degradation in long-context reasoning tasks. Through extensive experiments on long-context QA benchmarks, we find our method to outperform competitive baselines by large margins, achieving at least 8.4 and 7.9 EM gains on the long-context versions of HotpotQA and SQuAD datasets, respectively.