# Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference arXiv: https://arxiv.org/abs/2408.08590 ยท HTML: https://mlsanity.com/paper/2408.08590/ ## Key facts - Cited by 29 papers in the map (data through 14 Oct 2026). - Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. - Builds on 8 of its 60 references that are in the map. - First posted on arXiv 16 Aug 2024. ## Placed in - LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/) - Mechanistic interpretability / [Steering and interpreting LLM reasoning](https://mlsanity.com/mechinterp/steering-and-interpreting-llm-reasoning/) ## Builds on (most cited first, 8 of 8) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020) - [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023) - [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021) - [How to use and interpret activation patching](https://mlsanity.com/paper/2404.15255/) (2024) - [GenericsKB: A Knowledge Base of Generic Statements](https://mlsanity.com/paper/2005.00660/) (2020) - [Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanism](https://mlsanity.com/paper/2405.15302/) (2024) ## Built on by (most cited first, 29 of 29) - [Attention Heads of Large Language Models: A Survey](https://mlsanity.com/paper/2409.03752/) (2024) - [Transformers Struggle to Learn to Search](https://mlsanity.com/paper/2412.04703/) (2024) - [Mitigating Content Effects on Reasoning in Language Models through Fine-Grained Activation Steering](https://mlsanity.com/paper/2505.12189/) (2025) - [A Implies B: Circuit Analysis in LLMs for Propositional Logical Reasoning](https://mlsanity.com/paper/2411.04105/) (2024) - [Understanding Synthetic Context Extension via Retrieval Heads](https://mlsanity.com/paper/2410.22316/) (2024) - [SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoning](https://mlsanity.com/paper/2410.14399/) (2024) - [Mechanistic Indicators of Understanding in Large Language Models](https://mlsanity.com/paper/2507.08017/) (2025) - [Signatures of human-like processing in Transformer forward passes](https://mlsanity.com/paper/2504.14107/) (2025) - [Emergence and Localisation of Semantic Role Circuits in LLMs](https://mlsanity.com/paper/2511.20910/) (2025) - [Compressed Sensing for Capability Localization in Large Language Models](https://mlsanity.com/paper/2603.03335/) (2026) - [When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitions](https://mlsanity.com/paper/2605.22873/) (2026) - [Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility](https://mlsanity.com/paper/2608.03930/) (2026) - [Deductive Logic in Language Models: Horizontal vs Vertical Reasoning](https://mlsanity.com/paper/2510.09340/) (2025) - [FregeLogic at SemEval 2026 Task 11: A Hybrid Neuro-Symbolic Architecture for Content-Robust Syllogistic Validity Prediction](https://mlsanity.com/paper/2604.18328/) (2026) - [Entropy-Gradient Inversion: Moving Toward Internal Mechanism of Large Reasoning Models](https://mlsanity.com/paper/2605.17770/) (2026) - [From Circuit Evidence to Mechanistic Theory: An Inductive Logic Approach](https://mlsanity.com/paper/2605.21303/) (2026) - [SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance](https://mlsanity.com/paper/2606.09157/) (2026) - [Test, then Route: How Language Models Execute In-Context Conditional Rules Across Models and Languages](https://mlsanity.com/paper/2608.04183/) (2026) - [CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2609.07183/) (2026) - [ResidualDroppath: Enhancing Feature Reuse over Residual Connections](https://mlsanity.com/paper/2411.09475/) (2024) - [ITLC at SemEval-2026 Task 11: Normalization and Deterministic Parsing for Formal Reasoning in LLMs](https://mlsanity.com/paper/2603.02676/) (2026) - [Revealing Algorithmic Deductive Circuits for Logical Reasoning](https://mlsanity.com/paper/2605.27824/) (2026) - [Reproducing and Stress-Testing Two Approaches to LLM Reasoning Reliability: Test-Time Probability Aggregation and Logic-Representation Editing](https://mlsanity.com/paper/2608.08514/) (2026) - [Are you Talking Logic to Me? Assessing Language Models Syllogistic Reasoning Capabilities](https://mlsanity.com/paper/2608.12374/) (2026) - [Align, Unify, Suppress, Route: A Coherentist View of Transformer Computation](https://mlsanity.com/paper/2608.22034/) (2026) - [A Circuit for Plural Reference: How LLMs Represent and Retrieve Singular and Plural Entities](https://mlsanity.com/paper/2609.03687/) (2026) - [Understanding and Exploiting Anisotropy in Post-Training](https://mlsanity.com/paper/2609.32792/) (2026) - [A Polyphonic Conception of AI Understanding](https://mlsanity.com/paper/2609.36079/) (2026) - [GFGE: Unifying Explainable AI Methods through an Interpretation Framework](https://mlsanity.com/paper/2610.05225/) (2026) ## Abstract (arXiv) Recent studies on reasoning in language models (LMs) have sparked a debate on whether they can learn systematic inferential principles or merely exploit superficial patterns in the training data. To understand and uncover the mechanisms adopted for formal reasoning in LMs, this paper presents a mechanistic interpretation of syllogistic inference. Specifically, we present a methodology for circuit discovery aimed at interpreting content-independent and formal reasoning mechanisms. Through two distinct intervention methods, we uncover a sufficient and necessary circuit involving middle-term suppression that elucidates how LMs transfer information to derive valid conclusions from premises. Furthermore, we investigate how belief biases manifest in syllogistic inference, finding evidence of partial contamination from additional attention heads responsible for encoding commonsense and contextualized knowledge. Finally, we explore the generalization of the discovered mechanisms across various syllogistic schemes, model sizes and architectures. The identified circuit is sufficient and necessary for syllogistic schemes on which the models achieve high accuracy (>60%), with compatible activation patterns across models of different families. Overall, our findings suggest that LMs learn transferable content-independent reasoning mechanisms, but that, at the same time, such mechanisms do not involve generalizable and abstract logical primitives, being susceptible to contamination by the same world knowledge acquired during pre-training.