Reasoning or a Semblance of it? A Diagnostic Study of Transitive Reasoning in LLMs
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Evaluating Large Language Models (LLMs) on reasoning benchmarks demonstrates their ability to solve compositional questions. However, little is known of whether these models engage in genuine logical reasoning or simply rely on implicit cues to generate answers. In this paper, we investigate the transitive reasoning capabilities of two distinct LLM architectures, LLaMA 2 and Flan-T5, by manipulating facts within two compositional datasets: QASC and Bamboogle. We controlled for potential cues that might influence the models' performance, including (a) word/phrase overlaps across sections of test input; (b) models' inherent knowledge during pre-training or fine-tuning; and (c) Named Entities. Our findings reveal that while both models leverage (a), Flan-T5 shows more resilience to experiments (b and c), having less variance than LLaMA 2. This suggests that models may develop an understanding of transitivity through fine-tuning on knowingly relevant datasets, a hypothesis we leave to future work.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 4 of its 34 references that are in the map. First posted on arXiv 26 Oct 2024.
Builds on
4 of 34 references.
References · 4
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
- 2023Larger language models do in-context learning differentlycited by 394
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study
- 2025Answer-Centric or Reasoning-Driven? Uncovering the Latent Memory Anchor in LLMs
- 2025Reasoning Capabilities and Invariability of Large Language Models
- 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning
- 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
- 2024Calibrating Reasoning in Language Models with Internal Consistency
- 2024Understanding and Patching Compositional Reasoning in LLMs
- 2024Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance