mlsanityData to 7 Oct 2026

Natural Language Reasoning, A Survey

arXiv 2303.14725 ↗First posted 26 Mar 2023cs.CLcited by 61

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

This survey paper proposes a clearer view of natural language reasoning in the field of Natural Language Processing (NLP), both conceptually and practically. Conceptually, we provide a distinct definition for natural language reasoning in NLP, based on both philosophy and NLP scenarios, discuss what types of tasks require reasoning, and introduce a taxonomy of reasoning. Practically, we conduct a comprehensive literature review on natural language reasoning in NLP, mainly covering classical logical reasoning, natural language inference, multi-hop question answering, and commonsense reasoning. The paper also identifies and views backward reasoning, a powerful paradigm for multi-step reasoning, and introduces defeasible reasoning as one of the most important future directions in natural language reasoning research. We focus on single-modality unstructured natural language text, excluding neuro-symbolic techniques and mathematical reasoning.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 159 of its 219 references that are in the map. First posted on arXiv 26 Mar 2023.

Builds on

159 of 219 references.

Closest ancestors · 20

Same direction, newest first

  1. 2023Logical Reasoning over Natural Language as Knowledge Representation: A Survey21 Mar
  2. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity8 Feb
  3. 2022Language Models as Inductive Reasoners21 Dec
  4. 2022LAMBADA: Backward Chaining for Automated Reasoning in Natural Language20 Dec
  5. 2022Towards Reasoning in Large Language Models: A Survey20 Dec
  6. 2022APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning19 Dec
  7. 2022MetaLogic: Logical Reasoning Explanations with Fine-Grained Structure22 Oct
  8. 2022Disentangling Reasoning Capabilities from Language Models with Compositional Reasoning Transformers20 Oct
  9. 2022Measuring and Narrowing the Compositionality Gap in Language Models7 Oct
  10. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  11. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep
  12. 2022Faithful Reasoning Using Large Language Models30 Aug
8 more
  1. 2022Multi-Step Deductive Reasoning Over Natural Language: An Empirical Study on Out-of-Distribution Generalisation28 Jul
  2. 2022Language models show human-like content effects on reasoning tasks14 Jul
  3. 2022On the Paradox of Learning to Reason from Data23 May
  4. 2022Interpretable Proof Generation via Iterative Backward Reasoning22 May
  5. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning19 May
  6. 2022AbductionRules: Training Transformers to Explain Unexpected Inputs23 Mar
  7. 2022FaiRR: Faithful and Robust Deductive Reasoning over Natural Language19 Mar
  8. 2022MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoning1 Mar

Other references · 139

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  13. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  14. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  15. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  16. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  17. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  18. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  19. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  20. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  21. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  22. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  23. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  24. 2022Large Language Models Can Self-Improvecited by 656
  25. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  26. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  27. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  28. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  29. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  30. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  31. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  32. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  33. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  34. 2022Large Language Models Are Reasoning Teacherscited by 318
  35. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  36. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  37. 2019Abductive Commonsense Reasoningcited by 300
  38. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291

50 most cited of 139 All 139 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 10; 2025: 30; 2026: 5.

61 papers.

Sort:

By area:

  1. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  2. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  3. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoningcited by 89
  4. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 84
  5. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  6. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  7. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyondcited by 65
  8. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Informationcited by 42
  9. 2023A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoningcited by 41
  10. 2023Forward-Backward Reasoning in Large Language Models for Mathematical Verificationcited by 39
  11. 2023Explainable Claim Verification via Knowledge-Grounded Reasoning with Large Language Modelscited by 37
  12. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
38 more
  1. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  2. 2025Introducing LongCat-Flash-Thinking: A Technical Reportcited by 21
  3. 2026Large Language Model Reasoning Failurescited by 20
  4. 2025Auditing Meta-Cognitive Hallucinations in Reasoning Large Language Modelscited by 19
  5. 2023Just Ask One More Time! Self-Agreement Improves Reasoning of Language Models in (Almost) All Scenarioscited by 18
  6. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 16
  7. 2025Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learningcited by 15
  8. 2023InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Modelscited by 14
  9. 2023Through the Lens of Core Competency: Survey on Evaluation of Large Language Modelscited by 12
  10. 2024Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understandingcited by 12
  11. 2025Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignmentcited by 12
  12. 2024Your Context Is Not an Array: Unveiling Random Access Limitations in Transformerscited by 11
  13. 2024SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoningcited by 9
  14. 2025LeanProgress: Guiding Search for Neural Theorem Proving via Proof Progress Predictioncited by 9
  15. 2025Climplicit: Climatic Implicit Embeddings for Global Ecological Taskscited by 9
  16. 2024EconNLI: Evaluating Large Language Models on Economics Reasoningcited by 8
  17. 2025Graph-Augmented Reasoning: Evolving Step-by-Step Knowledge Graph Retrieval for LLM Reasoningcited by 8
  18. 2025Data Efficacy for Language Model Trainingcited by 8
  19. 2023Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Modelscited by 7
  20. 2023Conditions for Length Generalization in Learning Reasoning Skillscited by 7
  21. 2024Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generationcited by 7
  22. 2025Orion-Bix: Bi-Axial Attention for Tabular In-Context Learningcited by 6
  23. 2025Auditing the Ethical Logic of Generative AI Modelscited by 5
  24. 2025CoRE: Enhancing Metacognition with Label-free Self-evaluation in LRMscited by 5
  25. 2023LogicNet: A Logical Consistency Embedded Face Attribute Learning Networkcited by 4
  26. 2024"Let's Argue Both Sides": Argument Generation Can Force Small Models to Utilize Previously Inaccessible Reasoning Capabilitiescited by 4
  27. 2025AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinkingcited by 4
  28. 2025Think How to Think: Mitigating Overthinking with Autonomous Difficulty Cognition in Large Reasoning Modelscited by 4
  29. 2025SR-FoT: A Syllogistic-Reasoning Framework of Thought for Large Language Models Tackling Knowledge-based Reasoning Taskscited by 3
  30. 2025An Empirical Study of Conformal Prediction in LLM with ASP Scaffolds for Robust Reasoningcited by 3
  31. 2025Correlation or Causation: Analyzing the Causal Structures of LLM and LRM Reasoning Processcited by 3
  32. 2025ReTrace: Interactive Visualizations for Reasoning Traces of Large Reasoning Modelscited by 3
  33. 2025Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboardcited by 2
  34. 2025Enabling On-Device Medical AI Assistants via Input-Driven Saliency Adaptationcited by 2
  35. 2025LogiNumSynth: Synthesizing Joint Logical-Numerical Reasoning Problems for Language Modelscited by 2
  36. 2025LLM-based Automated Theorem Proving Hinges on Scalable Synthetic Data Generationcited by 1
  37. 2025Quality over Quantity: An Effective Large-Scale Data Reduction Strategy Based on Pointwise V-Informationcited by 1
  38. 2025How Well Do LLMs Understand Drug Mechanisms? A Knowledge + Reasoning Evaluation Datasetcited by 1

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Reliable Reasoning Beyond Natural Language
  2. 2022A Neural-Symbolic Approach to Natural Language Understanding
  3. 2025Reasoning-Grounded Natural Language Explanations for Language Models
  4. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
  5. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  6. 2023Explainable Verbal Reasoner Plus (EVR+): A Natural Language Reasoning Framework that Supports Diverse Compositional Reasoning
  7. 2023CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks
  8. 2022Can Transformers Reason in Fragments of Natural Language?