# Transformer See, Transformer Do: Copying as an Intermediate Step in Learning Analogical Reasoning

arXiv: https://arxiv.org/abs/2604.06501 · HTML: https://mlsanity.com/paper/2604.06501/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 7 Apr 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 25 of 25)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity](https://mlsanity.com/paper/2506.06941/) (2025)
- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025)
- [ARC Prize 2024: Technical Report](https://mlsanity.com/paper/2412.04604/) (2024)
- [Language Models Implement Simple Word2Vec-style Vector Arithmetic](https://mlsanity.com/paper/2305.16130/) (2023)
- [Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond](https://mlsanity.com/paper/2306.09841/) (2023)
- [Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices](https://mlsanity.com/paper/2008.02790/) (2020)
- [Combining Induction and Transduction for Abstract Reasoning](https://mlsanity.com/paper/2411.02272/) (2024)
- [Meta-learning in natural and artificial intelligence](https://mlsanity.com/paper/2011.13464/) (2020)
- [Evaluating the Robustness of Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2411.14215/) (2024)
- [Climbing the Ladder of Reasoning: What LLMs Can-and Still Can't-Solve after SFT?](https://mlsanity.com/paper/2504.11741/) (2025)
- [Few-shot Visual Reasoning with Meta-analogical Contrastive Learning](https://mlsanity.com/paper/2007.12020/) (2020)
- [Response: Emergent analogical reasoning in large language models](https://mlsanity.com/paper/2308.16118/) (2023)
- [Do large language models solve verbal analogies like children do?](https://mlsanity.com/paper/2310.20384/) (2023)
- [Can Large Language Models generalize analogy solving like children can?](https://mlsanity.com/paper/2411.02348/) (2024)
- [DeepGAR: Deep Graph Learning for Analogical Reasoning](https://mlsanity.com/paper/2211.10821/) (2022)
- [Modelling Analogies and Analogical Reasoning: Connecting Cognitive Science Theory and NLP Research](https://mlsanity.com/paper/2509.09381/) (2025)
- [Enhancing Analogical Reasoning in the Abstraction and Reasoning Corpus via Model-Based RL](https://mlsanity.com/paper/2408.14855/) (2024)
- [Propositional Logic for Probing Generalization in Neural Networks](https://mlsanity.com/paper/2506.08978/) (2025)

## Abstract (arXiv)

Analogical reasoning is a hallmark of human intelligence, enabling us to solve new problems by transferring knowledge from one situation to another. Yet, developing artificial intelligence systems capable of robust human-like analogical reasoning has proven difficult. In this work, we train transformers using Meta-Learning for Compositionality (MLC) on an analogical reasoning task (letter-string analogies) and assess their generalization capabilities. We find that letter-string analogies become learnable when guiding the models to attend to the most informative problem elements, induced by including copy tasks in the training data. Furthermore, generalization to new alphabets improves when models are trained with more heterogeneous datasets. For the best training run, our 3-layer encoder-decoder model performs on par with frontier models on our letter-string analogy datasets. The MLC approach also enables some generalization to compositions of trained transformations, but not to completely novel transformations. To understand how the model solves the analogies, we identify an algorithm for one analogy task that approximates the model's computations. We verify this using interpretability analyses and show that head-level attention weights causally affect model output, allowing steering between different analogy tasks. We discuss the implications of our findings for generalization capabilities of larger models and the parallels to human analogical reasoning.