# Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model

arXiv: https://arxiv.org/abs/2212.09146 · HTML: https://mlsanity.com/paper/2212.09146/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 18 Dec 2022.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [QASC: A Dataset for Question Answering via Sentence Composition](https://mlsanity.com/paper/1910.11473/) (2019)
- [Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval](https://mlsanity.com/paper/2009.12756/) (2020)
- [Explaining Answers with Entailment Trees](https://mlsanity.com/paper/2104.08661/) (2021)
- [Faithful Reasoning Using Large Language Models](https://mlsanity.com/paper/2208.14271/) (2022)
- [Training Language Models with Memory Augmentation](https://mlsanity.com/paper/2205.12674/) (2022)
- [Answering Open-Domain Questions of Varying Reasoning Steps from Text](https://mlsanity.com/paper/2010.12527/) (2020)
- [Natural Language Deduction through Search over Statement Compositions](https://mlsanity.com/paper/2201.06028/) (2022)

## Built on by (most cited first, 30 of 32)

- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [Query Rewriting for Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2305.14283/) (2023)
- [Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey](https://mlsanity.com/paper/2311.07914/) (2023)
- [ReasonIR: Training Retrievers for Reasoning Tasks](https://mlsanity.com/paper/2504.20595/) (2025)
- [Reliable, Adaptable, and Attributable Language Models with Retrieval](https://mlsanity.com/paper/2403.03187/) (2024)
- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks](https://mlsanity.com/paper/2305.18395/) (2023)
- [Scaling Retrieval-Based Language Models with a Trillion-Token Datastore](https://mlsanity.com/paper/2407.12854/) (2024)
- [BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information](https://mlsanity.com/paper/2306.07934/) (2023)
- [How Easily do Irrelevant Inputs Skew the Responses of Large Language Models?](https://mlsanity.com/paper/2404.03302/) (2024)
- [Are LLMs Ready for Real-World Materials Discovery?](https://mlsanity.com/paper/2402.05200/) (2024)
- [Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence](https://mlsanity.com/paper/2503.05037/) (2025)
- [How Well Do Large Language Models Truly Ground?](https://mlsanity.com/paper/2311.09069/) (2023)
- [RuleRAG: Rule-Guided Retrieval-Augmented Generation with Language Models for Question Answering](https://mlsanity.com/paper/2410.22353/) (2024)
- [Dense Passage Retrieval: Is it Retrieving?](https://mlsanity.com/paper/2402.11035/) (2024)
- [ArgRAG: Explainable Retrieval Augmented Generation using Quantitative Bipolar Argumentation](https://mlsanity.com/paper/2508.20131/) (2025)
- [On Evaluating the Integration of Reasoning and Action in LLM Agents with Database Question Answering](https://mlsanity.com/paper/2311.09721/) (2023)
- [Tell me what I need to know: Exploring LLM-based (Personalized) Abstractive Multi-Source Meeting Summarization](https://mlsanity.com/paper/2410.14545/) (2024)
- [Adapting to Non-Stationary Environments: Multi-Armed Bandit Enhanced Retrieval-Augmented Generation on Knowledge Graphs](https://mlsanity.com/paper/2412.07618/) (2024)
- [Controlled Retrieval-augmented Context Evaluation for Long-form RAG](https://mlsanity.com/paper/2506.20051/) (2025)
- [Great Memory, Shallow Reasoning: Limits of kNN-LMs](https://mlsanity.com/paper/2408.11815/) (2024)
- [Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarks](https://mlsanity.com/paper/2507.01297/) (2025)
- [AQA: Adaptive Question Answering in a Society of LLMs via Contextual Multi-Armed Bandit](https://mlsanity.com/paper/2409.13447/) (2024)
- [R^2AG: Incorporating Retrieval Information into Retrieval Augmented Generation](https://mlsanity.com/paper/2406.13249/) (2024)
- [SEGMENT+: Long Text Processing with Short-Context Language Models](https://mlsanity.com/paper/2410.06519/) (2024)
- [RAG over Thinking Traces Can Improve Reasoning Tasks](https://mlsanity.com/paper/2605.03344/) (2026)
- [Unveiling the Magic: Investigating Attention Distillation in Retrieval-augmented Generation](https://mlsanity.com/paper/2402.11794/) (2024)
- [When Life gives you LLMs, make LLM-ADE: Large Language Models with Adaptive Data Engineering](https://mlsanity.com/paper/2404.13028/) (2024)
- [Oreo: A Plug-in Context Reconstructor to Enhance Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.13019/) (2025)

## Abstract (arXiv)

Augmenting pretrained language models with retrievers has shown promise in effectively solving common NLP problems, such as language modeling and question answering. In this paper, we evaluate the strengths and weaknesses of popular retriever-augmented language models, namely kNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and Contriever + Flan-T5, in reasoning over retrieved statements across different tasks. Our findings indicate that the simple similarity metric employed by retrievers is insufficient for retrieving all the necessary statements for reasoning. Additionally, the language models do not exhibit strong reasoning even when provided with only the required statements. Furthermore, when combined with imperfect retrievers, the performance of the language models becomes even worse, e.g., Flan-T5's performance drops by 28.6% when retrieving 5 statements using Contriever. While larger language models improve performance, there is still a substantial room for enhancement. Our further analysis indicates that multihop retrieve-and-read is promising for large language models like GPT-3.5, but does not generalize to other language models like Flan-T5-xxl.