mlsanityData to 14 Oct 2026

Instantiation-based Formalization of Logical Reasoning Tasks using Language Models and Logical Solvers

arXiv 2501.16961 ↗First posted 28 Jan 2025cs.AIcited by 7

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Robustness of reasoning remains a significant challenge for large language models, and addressing it is essential for the practical applicability of AI-driven reasoning systems. We introduce Semantic Self-Verification (SSV), a novel approach that addresses the key challenge in combining language models with the rigor of logical solvers: to accurately formulate the reasoning problem from natural language to the formal language of the solver. SSV uses a consistency-based approach to produce strong abstract formalizations of problems using concrete instantiations that are generated by the model and verified by the solver. In addition to significantly advancing the overall reasoning accuracy over the state-of-the-art, a key novelty that this approach presents is a feature of verification that has near-perfect precision over a significant coverage of cases, as we demonstrate on open reasoning benchmarks. We propose such *near-certain reasoning* as a new approach to reduce the need for manual verification in many cases, taking us closer to more dependable and autonomous AI reasoning systems.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 30 of its 40 references that are in the map. First posted on arXiv 28 Jan 2025.

Builds on

30 of 40 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024LLM-ARC: Enhancing LLMs with an Automated Reasoning Critic25 Jun
  2. 2023Deductive Verification of Chain-of-Thought Reasoning6 Jun
  3. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning20 May
  4. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  5. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep
  6. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning19 May

Other references · 24

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  7. 2023Teaching Large Language Models to Self-Debugcited by 988
  8. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  9. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
  10. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 324
  11. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  12. 2022Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofscited by 291
12 more
  1. 2020Transformers as Soft Reasoners over Languagecited by 270
  2. 2022Autoformalization with Large Language Modelscited by 263
  3. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 207
  4. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 196
  5. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 194
  6. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 159
  7. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  8. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 108
  9. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  10. 2022Generating Natural Language Proofs with Verifier-Guided Searchcited by 52
  11. 2022Entailer: Answering Questions with Faithful and Truthful Chains of Reasoningcited by 38
  12. 2023Thought Propagation: An Analogical Approach to Complex Reasoning with Large Language Modelscited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025Autoformalization in the Era of Large Language Models: A Surveycited by 21
  3. 2025Faithful and Robust LLM-Driven Theorem Proving for NLI Explanationscited by 8
  4. 2025Step-Wise Formal Verification for LLM-Based Mathematical Problem Solvingcited by 5
  5. 2026Neuro-Symbolic Compliance: Integrating LLMs and SMT Solvers for Automated Financial Legal Analysiscited by 3
  6. 2026VeriSimpl: Robust Optimization Modeling from Natural Language using Simplification-based Verificationcited by 2
  7. 2025Reasoning Systems as Structured Processes: Foundations, Failures, and Formal Criteriacited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning
  2. 2026Draft-and-Prune: Improving the Reliability of Auto-formalization for Logical Reasoning
  3. 2025NP-Hard Lower Bound Complexity for Semantic Self-Verification
  4. 2026Sequent-Prover: Training Agents for Formal, Checkable SMT-based Reasoning
  5. 2024Automated Theorem Provers Help Improve Large Language Model Reasoning
  6. 2026VeryTrace: Verifying Reasoning Traces through Compilable Formalism and Structured Verification
  7. 2026Self-Supervised Theorem Discovery in a Formal Axiomatic System
  8. 2026PrologMCP: A Standardized Prolog Tool Interface for LLM Agents