mlsanityData to 7 Oct 2026

Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning

arXiv 2205.09712 ↗First posted 19 May 2022cs.AI cs.CLcited by 331

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have been shown to be capable of impressive few-shot generalisation to new tasks. However, they still tend to perform poorly on multi-step logical reasoning problems. Here we carry out a comprehensive evaluation of LLMs on 50 tasks that probe different aspects of logical reasoning. We show that language models tend to perform fairly well at single step inference or entailment tasks, but struggle to chain together multiple reasoning steps to solve more complex problems. In light of this, we propose a Selection-Inference (SI) framework that exploits pre-trained LLMs as general processing modules, and alternates between selection and inference to generate a series of interpretable, casual reasoning steps leading to the final answer. We show that a 7B parameter LLM used within the SI framework in a 5-shot generalisation setting, with no fine-tuning, yields a performance improvement of over 100% compared to an equivalent vanilla baseline on a suite of 10 logical reasoning tasks. The same model in the same setting even outperforms a significantly larger 280B parameter baseline on the same suite of tasks. Moreover, answers produced by the SI framework are accompanied by a causal natural-language-based reasoning trace, which has important implications for the safety and trustworthiness of the system.

Key facts. Cited by 331 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 32 of its 52 references that are in the map. First posted on arXiv 19 May 2022.

Builds on

32 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?25 Feb
  2. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  6. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  9. 2015Neural Module Networkscited by 375
  10. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 369
  11. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  12. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
18 more
  1. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  2. 2022Teaching language models to support answers with verified quotescited by 290
  3. 2020Transformers as Soft Reasoners over Languagecited by 265
  4. 2020Neurosymbolic AI: The 3rd Wavecited by 253
  5. 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 194
  6. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  7. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  8. 2021Explaining Answers with Entailment Treescited by 148
  9. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  10. 2019Learning by Abstraction: The Neural State Machinecited by 113
  11. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  12. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  13. 2019Neural Module Networks for Reasoning over Textcited by 65
  14. 2020PRover: Proof Generation for Interpretable Reasoning over Rulescited by 53
  15. 2020Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-Answeringcited by 49
  16. 2021Tell me why! Explanations support learning relational and causal structurecited by 36
  17. 2022Natural Language Deduction through Search over Statement Compositionscited by 35
  18. 2020Critical Thinking for Language Modelscited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 28; 2023: 118; 2024: 81; 2025: 73; 2026: 31.

331 papers.

Sort:

By area:

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Let's Verify Step by Stepcited by 3,598
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  5. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  6. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  7. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  8. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  9. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  10. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  11. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  12. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
38 more
  1. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  2. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  3. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  4. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  5. 2023Cognitive Architectures for Language Agentscited by 422
  6. 2023Augmented Language Models: a Surveycited by 398
  7. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  8. 2023Language Models can Solve Computer Taskscited by 371
  9. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  10. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  11. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  12. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  13. 2023Tool Learning with Foundation Modelscited by 302
  14. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  15. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  16. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  17. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  18. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 249
  19. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 233
  20. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  21. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  22. 2023Large Language Models as General Pattern Machinescited by 196
  23. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 180
  24. 2023Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoningcited by 177
  25. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  26. 2022Talking About Large Language Modelscited by 174
  27. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  28. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 163
  29. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 161
  30. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  31. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  32. 2023Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Frameworkcited by 139
  33. 2022Faithful Reasoning Using Large Language Modelscited by 130
  34. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 122
  35. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  36. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  37. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoningcited by 108
  38. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 107

50 most cited of 331 All 331 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Logical Reasoning in Large Language Models: A Survey
  2. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  3. 2024Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approach
  4. 2025Selection, Reflection and Self-Refinement: Revisit Reasoning Tasks via a Causal Lens
  5. 2025Can Large Language Models Learn Formal Logic? A Data-Driven Training and Evaluation Framework
  6. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
  7. 2025Efficient Inference for Large Reasoning Models: A Survey
  8. 2025Do Large Language Models Excel in Complex Logical Reasoning with Formal Language?