When Models Know More Than They Say: Probing Analogical Reasoning in LLMs
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Analogical reasoning is a core cognitive faculty essential for narrative understanding. While LLMs perform well when surface and structural cues align, they struggle in cases where an analogy is not apparent on the surface but requires latent information, suggesting limitations in abstraction and generalisation. In this paper we compare a model's probed representations with its prompted performance at detecting narrative analogies, revealing an asymmetry: for rhetorical analogies, probing significantly outperforms prompting in open-source models, while for narrative analogies, they achieve a similar (low) performance. This suggests that the relationship between internal representations and prompted behavior is task-dependent and may reflect limitations in how prompting accesses available information.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 10 of its 30 references that are in the map. First posted on arXiv 4 Apr 2026.
Builds on
10 of 30 references.
References · 10
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
- 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
- 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 171
- 2023Large Language Models as Analogical Reasonerscited by 107
- 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
- 2019An Annotated Dataset of Coreference in English Literaturecited by 58
- 2024Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models using Minimal Pairscited by 24
- 2020Measuring Information Propagation in Literary Social Networkscited by 15
- 2025NarraBench: A Comprehensive Framework for Narrative Benchmarkingcited by 13
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025The Curious Case of Analogies: Investigating Analogical Reasoning in Large Language Models
- 2025Modeling Understanding of Story-Based Analogies Using Large Language Models
- 2024LLMs as Models for Analogical Reasoning
- 2026Enhancing Structural Mapping with LLM-derived Abstractions for Analogical Reasoning in Narratives
- 2023ARN: Analogical Reasoning on Narratives
- 2024AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies
- 2022Understanding Narratives through Dimensions of Analogy
- 2023Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction