mlsanityData to 14 Oct 2026

Are you Talking Logic to Me? Assessing Language Models Syllogistic Reasoning Capabilities

arXiv 2608.12374 ↗First posted 22 Jul 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Language models (LMs) struggle with logical tasks like reasoning on syllogisms. It has been shown that Knowledge Representation (KR) plays a crucial role in expressing input information to help models solve tasks. This observation motivates our study of the impact of different formal KR notations on syllogistic reasoning by extending the FOLIO and P-FOLIO datasets. Our experiments on Small Language Models (SLMs) in Supervised Fine-Tuning (SFT) and Zero-Shot (ZS) settings show that the choice of input notation can yield performances competitive with natural language while enabling faster inference. We also propose a syllogistic categorization method (SEF) and use it to enrich ZS prompts with logical definitions, which boost reasoning in small models. We open-source our framework, Common Logic Grammar Construction (CLGC), as the first Python library for automatically generating syllogisms in KR notations and defining their SEF categories.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 24 of its 38 references that are in the map. First posted on arXiv 22 Jul 2026.

Builds on

24 of 38 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026Abstract Activation Spaces for Content-Invariant Reasoning in Large Language Models2 Feb
  2. 2025Improving Chain-of-Thought Reasoning via Quasi-Symbolic Abstractions18 Feb
  3. 2025Investigating the Robustness of Deductive Reasoning with Large Language Models4 Feb
  4. 2025JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models24 Jan
  5. 2024P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chains11 Oct
  6. 2024Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference16 Aug
  7. 2024Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset8 Aug
  8. 2024A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences17 Jun
  9. 2023A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models1 Nov
  10. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  11. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep
  12. 2022Language models show human-like content effects on reasoning tasks14 Jul

Other references · 12

  1. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 324
  2. 2023Faithful Chain-of-Thought Reasoningcited by 288
  3. 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 127
  4. 2024Faithful Logical Reasoning via Symbolic Chain-of-Thoughtcited by 100
  5. 2014Clingo = ASP + Control: Preliminary Reportcited by 56
  6. 2024Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Provingcited by 35
  7. 2025Mitigating Content Effects on Reasoning in Language Models through Fine-Grained Activation Steeringcited by 18
  8. 2023Evaluating the Deductive Competence of Large Language Modelscited by 12
  9. 2024SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoningcited by 9
  10. 2025LogicQA: Logical Anomaly Detection with Vision Language Model Generated Questionscited by 9
  11. 2025Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineeringcited by 2
  12. 2026SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performancecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Hybrid Models for Natural Language Reasoning: The Case of Syllogistic Logic
  2. 2026Do Diagrams Help Large Language Models Reason? Evidence from Syllogistic Reasoning
  3. 2025Understanding Syllogistic Reasoning in LLMs from Formal and Natural Language Perspectives
  4. 2026A Syllogistic Probe: Tracing the Evolution of Logic Reasoning in Large Language Models
  5. 2024Categorical Syllogisms Revisited: A Review of the Logical Reasoning Abilities of LLMs for Analyzing Categorical Syllogism
  6. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models
  7. 2025Teaching Small Language Models to Learn Logic through Meta-Learning
  8. 2024Proceedings of the First International Workshop on Next-Generation Language Models for Knowledge Representation and Reasoning (NeLaMKRR 2024)