# Language Models can be Logical Solvers

arXiv: https://arxiv.org/abs/2311.06158 · HTML: https://mlsanity.com/paper/2311.06158/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 10 Nov 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 22 of 22)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning](https://mlsanity.com/paper/2309.05653/) (2023)
- [Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://mlsanity.com/paper/2305.12295/) (2023)
- [PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change](https://mlsanity.com/paper/2206.10498/) (2022)
- [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://mlsanity.com/paper/2210.01240/) (2022)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [Faithful Chain-of-Thought Reasoning](https://mlsanity.com/paper/2301.13379/) (2023)
- [Transformers as Soft Reasoners over Language](https://mlsanity.com/paper/2002.05867/) (2020)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [Deductive Verification of Chain-of-Thought Reasoning](https://mlsanity.com/paper/2306.03872/) (2023)
- [SatLM: Satisfiability-Aided Language Models Using Declarative Prompting](https://mlsanity.com/paper/2305.09656/) (2023)
- [Generating Natural Language Proofs with Verifier-Guided Search](https://mlsanity.com/paper/2205.12443/) (2022)
- [Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning](https://mlsanity.com/paper/2210.12217/) (2022)
- [Fifty Years of Prolog and Beyond](https://mlsanity.com/paper/2201.10816/) (2022)

## Built on by (most cited first, 24 of 24)

- [Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification](https://mlsanity.com/paper/2504.05419/) (2025)
- [VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models](https://mlsanity.com/paper/2504.15279/) (2025)
- [LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languages](https://mlsanity.com/paper/2405.13144/) (2024)
- [Puzzle Solving using Reasoning of Large Language Models: A Survey](https://mlsanity.com/paper/2402.11291/) (2024)
- [PREGO: online mistake detection in PRocedural EGOcentric videos](https://mlsanity.com/paper/2404.01933/) (2024)
- [Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study](https://mlsanity.com/paper/2506.04810/) (2025)
- [Can LLMs Reason in the Wild with Programs?](https://mlsanity.com/paper/2406.13764/) (2024)
- [LLM-SQL-Solver: Can LLMs Determine SQL Equivalence?](https://mlsanity.com/paper/2312.10321/) (2023)
- [TI-PREGO: Chain of Thought and In-Context Learning for Online Mistake Detection in PRocedural EGOcentric Videos](https://mlsanity.com/paper/2411.02570/) (2024)
- [PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion](https://mlsanity.com/paper/2403.03788/) (2024)
- [A Closer Look at Logical Reasoning with LLMs: The Choice of Tool Matters](https://mlsanity.com/paper/2406.00284/) (2024)
- [DiLA: Enhancing LLM Tool Learning with Differential Logic Layer](https://mlsanity.com/paper/2402.11903/) (2024)
- [Towards Logically Sound Natural Language Reasoning with Logic-Enhanced Language Model Agents](https://mlsanity.com/paper/2408.16081/) (2024)
- [Autoformalization of Game Descriptions using Large Language Models](https://mlsanity.com/paper/2409.12300/) (2024)
- [Generative Agents for Multi-Agent Autoformalization of Interaction Scenarios](https://mlsanity.com/paper/2412.08805/) (2024)
- [Can Language Models Pretend Solvers? Logic Code Simulation with LLMs](https://mlsanity.com/paper/2403.16097/) (2024)
- [Integrating Expert Knowledge into Logical Programs via LLMs](https://mlsanity.com/paper/2502.12275/) (2025)
- [Pyramid-Driven Alignment: Pyramid Principle Guided Integration of Large Language Models and Knowledge Graphs](https://mlsanity.com/paper/2410.12298/) (2024)
- [Throttling Web Agents Using Reasoning Gates](https://mlsanity.com/paper/2509.01619/) (2025)
- [Improving Factuality in LLMs via Inference-Time Knowledge Graph Construction](https://mlsanity.com/paper/2509.03540/) (2025)
- [One STEP at a time: Language Agents are Stepwise Planners](https://mlsanity.com/paper/2411.08432/) (2024)
- [Human-Aligned Bench: Fine-Grained Assessment of Reasoning Ability in MLLMs vs. Humans](https://mlsanity.com/paper/2505.11141/) (2025)
- [Can LLMs Solve ASP Problems? Insights from a Benchmarking Study (Extended Version)](https://mlsanity.com/paper/2507.19749/) (2025)
- [SCRuB: Social Concept Reasoning under Rubric-Based Evaluation](https://mlsanity.com/paper/2605.06444/) (2026)

## Abstract (arXiv)

Logical reasoning is a fundamental aspect of human intelligence and a key component of tasks like problem-solving and decision-making. Recent advancements have enabled Large Language Models (LLMs) to potentially exhibit reasoning capabilities, but complex logical reasoning remains a challenge. The state-of-the-art, solver-augmented language models, use LLMs to parse natural language logical questions into symbolic representations first and then adopt external logical solvers to take in the symbolic representations and output the answers. Despite their impressive performance, any parsing errors will inevitably result in the failure of the execution of the external logical solver and no answer to the logical questions. In this paper, we introduce LoGiPT, a novel language model that directly emulates the reasoning processes of logical solvers and bypasses the parsing errors by learning to strict adherence to solver syntax and grammar. LoGiPT is fine-tuned on a newly constructed instruction-tuning dataset derived from revealing and refining the invisible reasoning process of deductive solvers. Experimental results on two public deductive reasoning datasets demonstrate that LoGiPT outperforms state-of-the-art solver-augmented LMs and few-shot prompting methods on competitive LLMs like ChatGPT or GPT-4.