mlsanityData to 7 Oct 2026

LOGIC-LM++: Multi-Step Refinement for Symbolic Formulations

arXiv 2407.02514 ↗First posted 22 Jun 2024cs.LO cs.AI cs.CLcited by 14

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In this paper we examine the limitations of Large Language Models (LLMs) for complex reasoning tasks. Although recent works have started to employ formal languages as an intermediate representation for reasoning tasks, they often face challenges in accurately generating and refining these formal specifications to ensure correctness. To address these issues, this paper proposes Logic-LM++, an improvement on Logic-LM . It uses the ability of LLMs to do pairwise comparisons, allowing the evaluation of the refinements suggested by the LLM. The paper demonstrates that Logic-LM++ outperforms Logic-LM and other contemporary techniques across natural language reasoning tasks on three datasets, FOLIO, ProofWriter and AR-LSAT, with an average improvement of 18.5% on standard prompting, 12.3% on chain of thought prompting and 5% on Logic-LM.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 19 of its 31 references that are in the map. First posted on arXiv 22 Jun 2024.

Builds on

19 of 31 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Rethinking Generative Large Language Model Evaluation for Semantic Comprehension12 Mar
  2. 2023Code Llama: Open Foundation Models for Code24 Aug
  3. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning20 May
  4. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Prompting16 May
  5. 2023STREET: A Multi-Task Structured Reasoning and Explanation Benchmark13 Feb

Other references · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  8. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  9. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  10. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  11. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  12. 2022Generating Sequences by Learning to Self-Correctcited by 271
2 more
  1. 2023Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?cited by 53
  2. 2021AR-LSAT: Investigating Analytical Reasoning of Textcited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 5.

14 papers.

Sort:

By area:

  1. 2024Divide and Translate: Compositional First-Order Logic Translation and Verification for Complex Logical Reasoningcited by 27
  2. 2025Faithful and Robust LLM-Driven Theorem Proving for NLI Explanationscited by 8
  3. 2025MuSLR: Multimodal Symbolic Logical Reasoningcited by 7
  4. 2025Are LLMs Stable Formal Logic Translators in Logical Reasoning Across Linguistically Diversified Texts?cited by 5
  5. 2024STACKFEED: Structured Textual Actor-Critic Knowledge Base Editing with FeedBackcited by 4
  6. 2024DANA: Domain-Aware Neurosymbolic Agents for Consistency and Accuracycited by 2
  7. 2026Doc2Spec: Synthesizing Formal Programming Specifications from Natural Language via Grammar Inductioncited by 2
  8. 2025FSLI: An Interpretable Formal Semantic System for One-Dimensional Ordering Inferencecited by 0
  9. 2025Towards a Neurosymbolic Reasoning System Grounded in Schematic Representationscited by 0
  10. 2025Intermediate Languages Matter: Formal Languages and LLMs affect Neurosymbolic Reasoningcited by 0
  11. 2026Delta1 with LLM: symbolic and neural integration for credible and explainable reasoningcited by 0
  12. 2026UFAL-CUNI at SemEval-2026 Task 11: An Efficient Modular Neuro-symbolic Method for Syllogistic Reasoningcited by 0
2 more
  1. 2026Pairwise Logical Selection of Enthymeme Completions under Semantic-Link Uncertaintycited by 0
  2. 2026Natural Language to First-Order Logic LLM-based Autoformalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Do Large Language Models Excel in Complex Logical Reasoning with Formal Language?
  2. 2024A Closer Look at Logical Reasoning with LLMs: The Choice of Tool Matters
  3. 2026Beyond Surface Forms: Symbolic Edits as a Test for Logical Reasoning with LLMs
  4. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models
  5. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
  6. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus
  7. 2025Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study
  8. 2024Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approach