mlsanityData to 7 Oct 2026

Feature Resemblance: Towards a Theoretical Understanding of Analogical Reasoning in Transformers

arXiv 2603.05143 ↗First posted 5 Mar 2026cs.CL cs.LGcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Understanding reasoning in large language models is complicated by evaluations that conflate multiple reasoning types. We isolate analogical reasoning, where a model transfers an attribute between entities that share known properties, and study when such transfer can emerge from training. To make the problem analytically tractable, we study a minimal transformer-style abstraction that isolates how learned representations support analogical reasoning. Within this setting, we prove three key results. First, joint training on similarity and attribution premises enables analogical reasoning through aligned representations. Second, sequential training succeeds only when similarity structure is learned before specific attributes, revealing a curriculum asymmetry. Third, in our stylized setting, two-hop reasoning (a → b, b → c Rightarrow a → c) can be viewed as analogical reasoning with identity bridges (b=b), which appear explicitly in training data. Together, these results reveal a unified mechanism: entities with shared properties become aligned in representation space, enabling property transfer through feature resemblance. Experiments with architectures up to 8B parameters show qualitative agreement with the theory and suggest that representational geometry plays an important role in analogical reasoning beyond the stylized model.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 33 of its 37 references that are in the map. First posted on arXiv 5 Mar 2026.

Builds on

33 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers12 Jun

Other references · 32

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  9. 2022In-context Learning and Induction Headscited by 930
  10. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  11. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  12. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
20 more
  1. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  2. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  3. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  4. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  5. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  6. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  7. 2016Deep Learning without Poor Local Minimacited by 233
  8. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  9. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
  10. 2016Identity Matters in Deep Learningcited by 107
  11. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  12. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 64
  13. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  14. 2024Understanding Factual Recall in Transformers via Associative Memoriescited by 45
  15. 2025Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thoughtcited by 32
  16. 2024Understanding Finetuning for Factual Knowledge Extractioncited by 31
  17. 2024Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamicscited by 27
  18. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 18
  19. 2024Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Factscited by 13
  20. 2025Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridgecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Emergent Analogical Reasoning in Transformers
  2. 2026Transformer See, Transformer Do: Copying as an Intermediate Step in Learning Analogical Reasoning
  3. 2023In-Context Analogical Reasoning with Pre-Trained Language Models
  4. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization
  5. 2025Analogical Reasoning Inside Large Language Models: Concept Vectors and the Limits of Abstraction
  6. 2026To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning
  7. 2025The Curious Case of Analogies: Investigating Analogical Reasoning in Large Language Models
  8. 2024Reasoning in Transformers -- Mitigating Spurious Correlations and Reasoning Shortcuts