mlsanityData to 7 Oct 2026

Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning

arXiv 2305.12295 ↗First posted 20 May 2023cs.CL cs.AIcited by 379

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models (LLMs) have shown human-like reasoning abilities but still struggle with complex logical problems. This paper introduces a novel framework, Logic-LM, which integrates LLMs with symbolic solvers to improve logical problem-solving. Our method first utilizes LLMs to translate a natural language problem into a symbolic formulation. Afterward, a deterministic symbolic solver performs inference on the formulated problem. We also introduce a self-refinement module, which utilizes the symbolic solver's error messages to revise symbolic formalizations. We demonstrate Logic-LM's effectiveness on five logical reasoning datasets: ProofWriter, PrOntoQA, FOLIO, LogicalDeduction, and AR-LSAT. On average, Logic-LM achieves a significant performance boost of 39.2% over using LLM alone with standard prompting and 18.4% over LLM with chain-of-thought prompting. Our findings suggest that Logic-LM, by combining LLMs with symbolic logic, offers a promising avenue for faithful logical reasoning. Code and data are publicly available at https://github.com/teacherpeterpan/Logic-LLM.

Key facts. Cited by 379 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 41 of its 67 references that are in the map. First posted on arXiv 20 May 2023.

Builds on

41 of 67 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Prompting16 May
  2. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-47 Apr
  3. 2023STREET: A Multi-Task Structured Reasoning and Explanation Benchmark13 Feb
  4. 2022Towards Reasoning in Large Language Models: A Survey20 Dec
  5. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning15 Dec
  6. 2022PAL: Program-aided Language Models18 Nov
  7. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  8. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep
  9. 2022NeuPSL: Neural Probabilistic Soft Logic27 May

Other references · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
20 more
  1. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  2. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  3. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  4. 2023Teaching Large Language Models to Self-Debugcited by 963
  5. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  6. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  7. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 369
  8. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  9. 2023Faithful Chain-of-Thought Reasoningcited by 285
  10. 2022Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofscited by 278
  11. 2020Transformers as Soft Reasoners over Languagecited by 265
  12. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  13. 2022Talking About Large Language Modelscited by 174
  14. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  15. 2021A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Levelcited by 96
  16. 2019Neural Module Networks for Reasoning over Textcited by 65
  17. 2015Hinge-Loss Markov Random Fields and Probabilistic Soft Logiccited by 63
  18. 2022Generating Natural Language Proofs with Verifier-Guided Searchcited by 52
  19. 2022Entailer: Answering Questions with Faithful and Truthful Chains of Reasoningcited by 38
  20. 2022Fifty Years of Prolog and Beyondcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 30; 2024: 101; 2025: 135; 2026: 113.

379 papers.

Sort:

By area:

  1. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  2. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  3. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  4. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  5. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  6. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  7. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  8. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  9. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 161
  10. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  11. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  12. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
38 more
  1. 2024SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinementcited by 124
  2. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  3. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  4. 2024Large Language Models Can Learn Temporal Reasoningcited by 110
  5. 2023GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoningcited by 107
  6. 2024Faithful Logical Reasoning via Symbolic Chain-of-Thoughtcited by 99
  7. 2023Learning From Mistakes Makes LLM Better Reasonercited by 97
  8. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 80
  9. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 80
  10. 2024SciAgent: Tool-augmented Language Models for Scientific Reasoningcited by 69
  11. 2024TableGPT2: A Large Multimodal Model with Tabular Data Integrationcited by 69
  12. 2024SynCode: LLM Generation with Grammar Augmentationcited by 66
  13. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyondcited by 65
  14. 2023Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translationcited by 59
  15. 2024Can Language Models Learn to Skip Steps?cited by 59
  16. 2023Grammar Prompting for Domain-Specific Language Generation with Large Language Modelscited by 58
  17. 2023Guiding Language Model Reasoning with Planning Tokenscited by 58
  18. 2024A Survey on Deep Learning for Theorem Provingcited by 55
  19. 2025Empowering LLMs with Logical Reasoning: A Comprehensive Surveycited by 51
  20. 2024Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Modelscited by 47
  21. 2024LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languagescited by 46
  22. 2024Puzzle Solving using Reasoning of Large Language Models: A Surveycited by 44
  23. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Informationcited by 42
  24. 2023BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biologycited by 42
  25. 2023A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoningcited by 41
  26. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  27. 2023Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Modelscited by 38
  28. 2025CRANE: Reasoning with constrained LLM generationcited by 38
  29. 2025LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Modelscited by 38
  30. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Promptscited by 36
  31. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
  32. 2024Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Provingcited by 35
  33. 2024System-1.x: Learning to Balance Fast and Slow Planning with Language Modelscited by 35
  34. 2024Planning Anything with Rigor: General-Purpose Zero-Shot Planning with LLM-based Formalized Programmingcited by 35
  35. 2023DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacycited by 34
  36. 2024How Likely Do LLMs with CoT Mimic Human Reasoning?cited by 34
  37. 2025Flow-of-Action: SOP Enhanced LLM-Based Multi-Agent System for Root Cause Analysiscited by 32
  38. 2025Semantic-Aware Logical Reasoning via a Semiotic Frameworkcited by 32

50 most cited of 379 All 379 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Closer Look at Logical Reasoning with LLMs: The Choice of Tool Matters
  2. 2026Beyond Surface Forms: Symbolic Edits as a Test for Logical Reasoning with LLMs
  3. 2026Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verification
  4. 2024Automated Theorem Provers Help Improve Large Language Model Reasoning
  5. 2025Adaptive LLM-Symbolic Reasoning via Dynamic Logical Solver Composition
  6. 2023Improved Logical Reasoning of Language Models via Differentiable Symbolic Programming
  7. 2025Adaptive Selection of Symbolic Languages for Improving LLM Logical Reasoning
  8. 2025Do Large Language Models Excel in Complex Logical Reasoning with Formal Language?