mlsanityData to 14 Oct 2026

Empowering LLMs with Logical Reasoning: A Comprehensive Survey

arXiv 2502.15652 ↗First posted 21 Feb 2025cs.AI cs.CLcited by 51

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have achieved remarkable successes on various tasks. However, recent studies have found that there are still significant challenges to the logical reasoning abilities of LLMs, which can be categorized into the following two aspects: (1) Logical question answering: LLMs often fail to generate the correct answer within a complex logical problem which requires sophisticated deductive, inductive or abductive reasoning given a collection of premises. (2) Logical consistency: LLMs are prone to producing responses contradicting themselves across different questions. For example, a state-of-the-art question-answering LLM Macaw, answers Yes to both questions Is a magpie a bird? and Does a bird have wings? but answers No to Does a magpie have wings?. To facilitate this research direction, we comprehensively investigate the most cutting-edge methods and propose a detailed taxonomy. Specifically, to accurately answer complex logic questions, previous methods can be categorized based on reliance on external solvers, prompts, and fine-tuning. To avoid logical contradictions, we discuss concepts and solutions of various logical consistencies, including implication, negation, transitivity, factuality consistencies, and their composites. In addition, we review commonly used benchmark datasets and evaluation metrics, and discuss promising research directions, such as extending to modal logic to account for uncertainty and developing efficient algorithms that simultaneously satisfy multiple logical consistencies.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 54 of its 67 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

54 of 67 references.

Closest ancestors · 30

Same direction, newest first

  1. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning3 Feb
  2. 2025VERUS-LM: a Versatile Framework for Combining LLMs with Symbolic Reasoning24 Jan
  3. 2024Aristotle: Mastering Logical Reasoning with A Logic-Complete Decompose-Search-Resolve Framework22 Dec
  4. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus19 Nov
  5. 2024Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approach29 Oct
  6. 2024Divide and Translate: Compositional First-Order Logic Translation and Verification for Complex Logical Reasoning10 Oct
  7. 2024LogicPro: Improving Complex Logical Reasoning via Program-Guided Learning19 Sep
  8. 2024On the Diagram of Thought16 Sep
  9. 2024Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset8 Aug
  10. 2024ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning14 Jul
  11. 2024Categorical Syllogisms Revisited: A Review of the Logical Reasoning Abilities of LLMs for Analyzing Categorical Syllogism26 Jun
  12. 2024Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models24 Jun
18 more
  1. 2024Scaling Synthetic Logical Reasoning Datasets with Context-Sensitive Declarative Grammars16 Jun
  2. 2024Are LLMs classical or nonmonotonic reasoners? Lessons from generics5 Jun
  3. 2024A Closer Look at Logical Reasoning with LLMs: The Choice of Tool Matters1 Jun
  4. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models23 Apr
  5. 2024DiLA: Enhancing LLM Tool Learning with Differential Logic Layer19 Feb
  6. 2024Conditional and Modal Reasoning in Large Language Models30 Jan
  7. 2024LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models1 Jan
  8. 2023DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacy28 Oct
  9. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers23 Oct
  10. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models2 Oct
  11. 2023Cumulative Reasoning with Large Language Models8 Aug
  12. 2023Exploring Self-supervised Logic-enhanced Training for Large Language Models23 May
  13. 2023Abstract Meaning Representation-Based Logic-Driven Data Augmentation for Logical Reasoning21 May
  14. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning20 May
  15. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Prompting16 May
  16. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-47 Apr
  17. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep
  18. 2022Multi-Step Deductive Reasoning Over Natural Language: An Empirical Study on Out-of-Distribution Generalisation28 Jul

Other references · 24

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  3. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  4. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 324
  5. 2023Faithful Chain-of-Thought Reasoningcited by 288
  6. 2020Transformers as Soft Reasoners over Languagecited by 270
  7. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 217
  8. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  9. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  10. 2024Faithful Logical Reasoning via Symbolic Chain-of-Thoughtcited by 100
  11. 2020Logic-Guided Data Augmentation and Regularization for Consistent Question Answeringcited by 78
  12. 2019A Logic-Driven Framework for Consistency of Neural Modelscited by 64
12 more
  1. 2021BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Beliefcited by 57
  2. 2021Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Textcited by 46
  3. 2022Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inferencecited by 40
  4. 2023Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Modelscited by 39
  5. 2022Entailer: Answering Questions with Faithful and Truthful Chains of Reasoningcited by 38
  6. 2023Consistency Analysis of ChatGPTcited by 37
  7. 2024Logically Consistent Language Models via Neuro-Symbolic Integrationcited by 30
  8. 2024Consistency Checks for Language Model Forecasterscited by 27
  9. 2024Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Modelscited by 23
  10. 2023Language Models with Rationalitycited by 15
  11. 2024Logical Consistency of Large Language Models in Fact-checkingcited by 15
  12. 2023In Search of the Long-Tail: Systematic Generation of Long-Tail Inferential Knowledge via Logical Rule Guided Searchcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 30; 2026: 21.

51 papers.

Sort:

By area:

  1. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 33
  2. 2025Mitigating Hallucinations in Large Language Models via Causal Reasoningcited by 13
  3. 2025Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoningcited by 9
  4. 2025What Breaks Knowledge Graph based RAG? Benchmarking and Empirical Insights into Reasoning under Incomplete Knowledgecited by 9
  5. 2025PEIRCE: Unifying Material and Formal Reasoning via LLM-Driven Neuro-Symbolic Refinementcited by 7
  6. 2025Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessmentcited by 7
  7. 2025Debunk and Infer: Multimodal Fake News Detection via Diffusion-Generated Evidence and LLM Reasoningcited by 7
  8. 2025Method-Based Reasoning for Large Language Models: Extraction, Reuse, and Continuous Improvementcited by 6
  9. 2025LogicReward: Incentivizing LLM Reasoning via Step-Wise Logical Supervisioncited by 6
  10. 2025Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Riskscited by 5
  11. 2025AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLPcited by 5
  12. 2025Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RLcited by 5
38 more
  1. 2025Conflict-Aware Fusion: Mitigating Logic Inertia in Large Language Models via Structured Cognitive Priorscited by 5
  2. 2026Towards Self-Robust LLMs: Intrinsic Prompt Noise Resistance via CoIPOcited by 5
  3. 2025Curious Causality-Seeking Agents Learn Meta Causal Worldcited by 4
  4. 2025Sound and Complete Neurosymbolic Reasoning with LLM-Grounded Interpretationscited by 4
  5. 2025Accelerating LLM Reasoning via Early Rejection with Partial Reward Modelingcited by 4
  6. 2025Do Language Models Follow Occam's Razor? An Evaluation of Parsimony in Inductive and Abductive Reasoningcited by 4
  7. 2025Question Answering with LLMs and Learning from Answer Setscited by 4
  8. 2025From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Trainingcited by 3
  9. 2025Normative Reasoning in Large Language Models: A Comparative Benchmark from Logical and Modal Perspectivescited by 3
  10. 2025Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysiscited by 2
  11. 2025Propose and Rectify: A Forensics-Driven MLLM Framework for Image Manipulation Localizationcited by 2
  12. 2025Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineeringcited by 2
  13. 2025Uncertainty-aware Reward Design Processcited by 1
  14. 2025SHERLOCK: Towards Dynamic Knowledge Adaptation in LLM-enhanced E-commerce Risk Managementcited by 1
  15. 2026An Agentic Framework for Neuro-Symbolic Programmingcited by 1
  16. 2026SiriusHelper: An LLM Agent-Based Operations Assistant for Big Data Platformscited by 1
  17. 2026FERA: Uncertainty-Aware Federated Reasoning for Large Language Modelscited by 1
  18. 2026REIN: Bridging the Gap between Reasoning and Reliability via Reflection and Abstention Alignmentcited by 1
  19. 2026LogicTrack: Auditing Reasoning Trajectories of Large Language Models with Formal Logic Solverscited by 1
  20. 2025SWI: Speaking with Intent in Large Language Modelscited by 0
  21. 2025Repairing Tool Calls Using Post-tool Execution Reflection and RAGcited by 0
  22. 2025Symbolic Neural Generation with Applications to Lead Discovery in Drug Designcited by 0
  23. 2025TCM-5CEval: Extended Deep Evaluation Benchmark for LLM's Comprehensive Clinical Research Competence in Traditional Chinese Medicinecited by 0
  24. 2025DETAIL Matters: Measuring the Impact of Prompt Specificity on Reasoning in Large Language Modelscited by 0
  25. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  26. 2026Logic-Parametric Neuro-Symbolic NLI: Controlling Logical Formalisms for Verifiable LLM Reasoningcited by 0
  27. 2026Rethinking the Flow-Based Gradual Domain Adaptation: A Semi-Dual Optimal Transport Perspectivecited by 0
  28. 2026FloCA: Towards Faithful and Logically Consistent Flowchart Reasoningcited by 0
  29. 2026Explicit Logic Channel for Validation and Enhancement of MLLMs on Zero-Shot Taskscited by 0
  30. 2026Discovering a Shared Logical Subspace: Steering LLM Logical Reasoning via Alignment of Natural-Language and Symbolic Viewscited by 0
  31. 2026SRTJ: Self-Evolving Rule-Driven Training-Free LLM Jailbreakingcited by 0
  32. 2026Towards Annotation-Free Validation of MLLMs: A Vision-Language Logical Consistency Metriccited by 0
  33. 2026Generating Graph-Like Logical Rules for Knowledge Graph Reasoning via Diffusion Modelscited by 0
  34. 2026HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimizationcited by 0
  35. 2026Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programmingcited by 0
  36. 2026PL-Guard: Probabilistic Logic Reasoning for LLM Guardrailscited by 0
  37. 2026Beyond Surface Forms: Symbolic Edits as a Test for Logical Reasoning with LLMscited by 0
  38. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
  2. 2025Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study
  3. 2025Do Large Language Models Excel in Complex Logical Reasoning with Formal Language?
  4. 2025Logical Reasoning in Large Language Models: A Survey
  5. 2025Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning
  6. 2025SATQuest: A Verifier for Logical Reasoning Evaluation and Reinforcement Fine-Tuning of LLMs
  7. 2022Faithful Reasoning Using Large Language Models
  8. 2024Critical-Questions-of-Thought: Steering LLM reasoning with Argumentative Querying