mlsanityData to 13 Oct 2026

Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study

arXiv 2506.04810 ↗First posted 5 Jun 2025cs.CL cs.AI cs.LOcited by 17

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Logical reasoning is a core capability for large language models (LLMs), yet existing benchmarks that rely solely on final-answer accuracy fail to capture the quality of the reasoning process. To address this, we introduce FineLogic, a fine-grained evaluation framework that assesses logical reasoning across three dimensions: overall accuracy, stepwise soundness, and representation-level probing. Leveraging this framework, we conduct a comprehensive study on how different supervision formats in fine-tuning shape reasoning abilities. We fine-tune LLMs on four supervision styles: one in natural language and three symbolic variants. We find a key trade-off: natural language supervision excels at generalization to out-of-distribution and long-chain problems, whereas symbolic supervision is superior at instilling structurally sound, atomic reasoning steps. Furthermore, our probing analysis indicates that fine-tuning primarily refines the model's step-by-step generation process, rather than improving its ability to converge on an answer early. Together, our framework and analysis provide a more rigorous lens for evaluating and improving logical reasoning in LLMs. The code is available at https://github.com/YujunZhou/FineLogic.

Key facts. Cited by 17 papers in the map (data through 13 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 33 of its 46 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

33 of 46 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Learning to Reason via Mixture-of-Thought for Logical Reasoning21 May
  2. 2025Stepwise Informativeness Search for Efficient and Effective LLM Reasoning21 Feb
  3. 2024On Memorization of Large Language Models in Logical Reasoning30 Oct
  4. 2024Divide and Translate: Compositional First-Order Logic Translation and Verification for Complex Logical Reasoning10 Oct
  5. 2024Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models24 Jun
  6. 2024Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models18 Jun
  7. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models23 Apr
  8. 2023Language Models can be Logical Solvers10 Nov
  9. 2023DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacy28 Oct
  10. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers23 Oct
  11. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models2 Oct
  12. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning20 May
5 more
  1. 2023ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness21 Apr
  2. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning15 Dec
  3. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  4. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep
  5. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning19 May

Other references · 16

  1. 2023GPT-4 Technical Reportcited by 18,278
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,247
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,595
  4. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 758
  5. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 313
  6. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 261
  7. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 103
  8. 2024Faithful Logical Reasoning via Symbolic Chain-of-Thoughtcited by 100
  9. 2023Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translationcited by 59
  10. 2022Generating Natural Language Proofs with Verifier-Guided Searchcited by 52
  11. 2025On the Trustworthiness of Generative Foundation Models: Guideline, Assessment, and Perspectivecited by 40
  12. 2024SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmarkcited by 35
4 more
  1. 2024Defending Jailbreak Prompts via In-Context Adversarial Gamecited by 29
  2. 2024Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?cited by 26
  3. 2024LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labscited by 24
  4. 2025AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Modelscited by 16

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 7; 2026: 10.

17 papers.

Sort:

By area:

  1. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 215
  2. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 59
  3. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  4. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Modelscited by 29
  5. 2026Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agentscited by 11
  6. 2025SLR: Automated Synthesis for Scalable Logical Reasoningcited by 6
  7. 2026Reasoning-Trace Collapse: Evaluating the Loss of Explicit Reasoning During Fine-Tuningcited by 4
  8. 2026ReEfBench: Quantifying the Reasoning Efficiency of LLMscited by 3
  9. 2026TDA-RC: Task-Driven Alignment for Knowledge-Based Reasoning Chains in Large Language Modelscited by 2
  10. 2026Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Datacited by 2
  11. 2025Quantifying LLM Biases Across Instruction Boundary in Mixed Question Formscited by 1
  12. 2025Concept Generalization in Humans and Large Language Models: Insights from the Number Gamecited by 1
5 more
  1. 2026Correct Prediction, Wrong Steps? Consensus Reasoning Knowledge Graph for Robust Chain-of-Thought Synthesiscited by 1
  2. 2026Discovering a Shared Logical Subspace: Steering LLM Logical Reasoning via Alignment of Natural-Language and Symbolic Viewscited by 0
  3. 2026SciR: A Controllable Benchmark for Scientific Reasoning in LLMscited by 0
  4. 2026GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMscited by 0
  5. 2026SAGE: Mitigating Long-Horizon Reasoning Biases via Topological Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
  2. 2026Beyond Surface Forms: Symbolic Edits as a Test for Logical Reasoning with LLMs
  3. 2024Reasoning or a Semblance of it? A Diagnostic Study of Transitive Reasoning in LLMs
  4. 2025Logical Reasoning in Large Language Models: A Survey
  5. 2025What Defines Good Reasoning in LLMs? Dissecting Reasoning Steps with Multi-Aspect Evaluation
  6. 2024A Closer Look at Logical Reasoning with LLMs: The Choice of Tool Matters
  7. 2025LogiDynamics: Unraveling the Dynamics of Inductive, Abductive and Deductive Logical Inferences in LLM Reasoning
  8. 2025SATQuest: A Verifier for Logical Reasoning Evaluation and Reinforcement Fine-Tuning of LLMs