# Are you Talking Logic to Me? Assessing Language Models Syllogistic Reasoning Capabilities

arXiv: https://arxiv.org/abs/2608.12374 · HTML: https://mlsanity.com/paper/2608.12374/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 22 Jul 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 24 of 24)

- [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://mlsanity.com/paper/2210.01240/) (2022)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [Faithful Chain-of-Thought Reasoning](https://mlsanity.com/paper/2301.13379/) (2023)
- [FOLIO: Natural Language Reasoning with First-Order Logic](https://mlsanity.com/paper/2209.00840/) (2022)
- [Language models show human-like content effects on reasoning tasks](https://mlsanity.com/paper/2207.07051/) (2022)
- [A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness](https://mlsanity.com/paper/2411.03350/) (2024)
- [Faithful Logical Reasoning via Symbolic Chain-of-Thought](https://mlsanity.com/paper/2405.18357/) (2024)
- [Clingo = ASP + Control: Preliminary Report](https://mlsanity.com/paper/1405.3694/) (2014)
- [Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving](https://mlsanity.com/paper/2405.01379/) (2024)
- [Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference](https://mlsanity.com/paper/2408.08590/) (2024)
- [JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models](https://mlsanity.com/paper/2501.14851/) (2025)
- [A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models](https://mlsanity.com/paper/2311.00445/) (2023)
- [Mitigating Content Effects on Reasoning in Language Models through Fine-Grained Activation Steering](https://mlsanity.com/paper/2505.12189/) (2025)
- [Improving Chain-of-Thought Reasoning via Quasi-Symbolic Abstractions](https://mlsanity.com/paper/2502.12616/) (2025)
- [P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chains](https://mlsanity.com/paper/2410.09207/) (2024)
- [Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset](https://mlsanity.com/paper/2408.04403/) (2024)
- [A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences](https://mlsanity.com/paper/2406.11341/) (2024)
- [Evaluating the Deductive Competence of Large Language Models](https://mlsanity.com/paper/2309.05452/) (2023)
- [SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoning](https://mlsanity.com/paper/2410.14399/) (2024)
- [LogicQA: Logical Anomaly Detection with Vision Language Model Generated Questions](https://mlsanity.com/paper/2503.20252/) (2025)
- [Investigating the Robustness of Deductive Reasoning with Large Language Models](https://mlsanity.com/paper/2502.04352/) (2025)
- [Abstract Activation Spaces for Content-Invariant Reasoning in Large Language Models](https://mlsanity.com/paper/2602.02462/) (2026)
- [Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineering](https://mlsanity.com/paper/2509.10249/) (2025)
- [SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance](https://mlsanity.com/paper/2606.09157/) (2026)

## Abstract (arXiv)

Language models (LMs) struggle with logical tasks like reasoning on syllogisms. It has been shown that Knowledge Representation (KR) plays a crucial role in expressing input information to help models solve tasks. This observation motivates our study of the impact of different formal KR notations on syllogistic reasoning by extending the FOLIO and P-FOLIO datasets. Our experiments on Small Language Models (SLMs) in Supervised Fine-Tuning (SFT) and Zero-Shot (ZS) settings show that the choice of input notation can yield performances competitive with natural language while enabling faster inference. We also propose a syllogistic categorization method (SEF) and use it to enrich ZS prompts with logical definitions, which boost reasoning in small models. We open-source our framework, Common Logic Grammar Construction (CLGC), as the first Python library for automatically generating syllogisms in KR notations and defining their SEF categories.