mlsanityData to 7 Oct 2026

Rethinking with Retrieval: Faithful Large Language Model Inference

arXiv 2301.00303 ↗First posted 31 Dec 2022cs.CL cs.AIcited by 145

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Despite the success of large language models (LLMs) in various natural language processing (NLP) tasks, the stored knowledge in these models may inevitably be incomplete, out-of-date, or incorrect. This motivates the need to utilize external knowledge to assist LLMs. Unfortunately, current methods for incorporating external knowledge often require additional training or fine-tuning, which can be costly and may not be feasible for LLMs. To address this issue, we propose a novel post-processing approach, rethinking with retrieval (RR), which retrieves relevant external knowledge based on the decomposed reasoning steps obtained from the chain-of-thought (CoT) prompting. This lightweight approach does not require additional training or fine-tuning and is not limited by the input length of LLMs. We evaluate the effectiveness of RR through extensive experiments with GPT-3 on three complex reasoning tasks: commonsense reasoning, temporal reasoning, and tabular reasoning. Our results show that RR can produce more faithful explanations and improve the performance of LLMs.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 39 of its 58 references that are in the map. First posted on arXiv 31 Dec 2022.

Builds on

39 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
26 more
  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  6. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  7. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  10. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  11. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  12. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  13. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  14. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  15. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  16. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  17. 2021Internet-Augmented Dialogue Generationcited by 217
  18. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 213
  19. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  20. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  21. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 87
  22. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  23. 2020INFOTABS: Inference on Tables as Semi-structured Datacited by 84
  24. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  25. 2021Incorporating External Knowledge to Enhance Tabular Reasoningcited by 23
  26. 2020Contextualized Representations Using Textual Encyclopedic Knowledgecited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 66; 2024: 48; 2025: 24; 2026: 7.

145 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  3. 2023Augmented Language Models: a Surveycited by 398
  4. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  5. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  6. 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoningcited by 326
  7. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  8. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  9. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  10. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  11. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  12. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
38 more
  1. 2024Language Models with Conformal Factuality Guaranteescited by 128
  2. 2023Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answeringcited by 97
  3. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  4. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  5. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  6. 2023Improving Language Models via Plug-and-Play Retrieval Feedbackcited by 87
  7. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  8. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  9. 2023Boosting Language Models Reasoning with Chain-of-Knowledge Promptingcited by 77
  10. 2023Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-Incited by 71
  11. 2023A Survey of Large Language Models Attributioncited by 71
  12. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  13. 2023Self-Knowledge Guided Retrieval Augmentation for Large Language Modelscited by 69
  14. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  15. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  16. 2024Building Guardrails for Large Language Modelscited by 61
  17. 2023PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptionscited by 53
  18. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Modelscited by 50
  19. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answeringcited by 49
  20. 2023Effective Large Language Model Adaptation for Improved Grounding and Citation Generationcited by 44
  21. 2023KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detectioncited by 43
  22. 2024Evaluating Language Model Agency through Negotiationscited by 43
  23. 2023"According to ...": Prompting Language Models Improves Quoting from Pre-Training Datacited by 42
  24. 2023Tab-CoT: Zero-shot Tabular Chain of Thoughtcited by 42
  25. 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 42
  26. 2023A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoningcited by 41
  27. 2024Training Language Models to Generate Text with Citations via Fine-grained Rewardscited by 40
  28. 2024FiDeLiS: Faithful Reasoning in Large Language Model for Knowledge Graph Question Answeringcited by 38
  29. 2023Re-Reading Improves Reasoning in Large Language Modelscited by 37
  30. 2024Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMscited by 34
  31. 2023Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operationcited by 33
  32. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  33. 2024Long2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recallcited by 32
  34. 2023Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Communitycited by 29
  35. 2024Source-Aware Training Enables Knowledge Attribution in Language Modelscited by 29
  36. 2023Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applicationscited by 28
  37. 2023Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Modelscited by 27
  38. 2023DRDT: Dynamic Reflection with Divergent Thinking for LLM-based Sequential Recommendationcited by 27

50 most cited of 145 All 145 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Disentangling Memory and Reasoning Ability in Large Language Models
  2. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Context
  3. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  4. 2026ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning
  5. 2025Reasoning on Multiple Needles In A Haystack
  6. 2025DeepRAG: Thinking to Retrieve Step by Step for Large Language Models
  7. 2026How Much Reasoning Do Retrieval-Augmented Models Add beyond LLMs? A Benchmarking Framework for Multi-Hop Inference over Hybrid Knowledge
  8. 2025LIR3AG: A Lightweight Rerank Reasoning Strategy Framework for Retrieval-Augmented Generation