mlsanityData to 7 Oct 2026

When can transformers reason with abstract symbols?

arXiv 2310.09753 ↗First posted 15 Oct 2023cs.CL cs.AI cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

We investigate the capabilities of transformer models on relational reasoning tasks. In these tasks, models are trained on a set of strings encoding abstract relations, and are then tested out-of-distribution on data that contains symbols that did not appear in the training dataset. We prove that for any relational reasoning task in a large family of tasks, transformers learn the abstract relations and generalize to the test set when trained by gradient descent on sufficiently large quantities of training data. This is in contrast to classical fully-connected networks, which we prove fail to learn to reason. Our results inspire modifications of the transformer architecture that add only two trainable parameters per head, and that we empirically demonstrate improve data efficiency for learning to reason.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 34 of its 55 references that are in the map. First posted on arXiv 15 Oct 2023.

Builds on

34 of 55 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  8. 2018On Lazy Training in Differentiable Programmingcited by 671
  9. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  10. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  11. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  12. 2017A simple neural network module for relational reasoningcited by 345
22 more
  1. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  2. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  3. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  4. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  5. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  6. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  7. 2017Recurrent Relational Networkscited by 71
  8. 2018Relational recurrent neural networkscited by 65
  9. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  10. 2023Mimetic Initialization of Self-Attention Layerscited by 52
  11. 2023Do Transformers Parse while Predicting the Masked Word?cited by 48
  12. 2020Emergent Symbols through Binding in External Memorycited by 47
  13. 2019Mean Field Analysis of Deep Neural Networkscited by 42
  14. 2020Learning Representations that Support Extrapolationcited by 36
  15. 2019An Explicitly Relational Neural Network Architecturecited by 35
  16. 2020Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets?cited by 32
  17. 2021Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalizationcited by 29
  18. 2016Distribution-Specific Hardness of Learning Neural Networkscited by 27
  19. 2022On Neural Architecture Inductive Biases for Relational Taskscited by 26
  20. 2023The Relational Bottleneck as an Inductive Bias for Efficient Abstractioncited by 26
  21. 2022On the non-universality of deep learning: quantifying the cost of symmetrycited by 17
  22. 2022An initial alignment between neural network and target is needed for gradient descent to learncited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 11; 2026: 4.

23 papers.

Sort:

By area:

  1. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  2. 2023Vamos: Versatile Action Models for Video Understandingcited by 40
  3. 2025FACTR: Force-Attending Curriculum Training for Contact-Rich Policy Learningcited by 31
  4. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  5. 2024What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Taskscited by 19
  6. 2024Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanismcited by 15
  7. 2024MLPs Learn In-Context on Regression and Classification Taskscited by 12
  8. 2025Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts?cited by 9
  9. 2024Transformer Normalisation Layers and the Independence of Semantic Subspacescited by 6
  10. 2024Interpret Your Decision: Logical Reasoning Regularization for Generalization in Visual Classificationcited by 5
  11. 2025Reason from Future: Reverse Thought Chain Enhances LLM Reasoningcited by 5
  12. 2025Learning to Add, Multiply, and Execute Algorithmic Instructions Exactly with Neural Networkscited by 4
11 more
  1. 2025Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspectivecited by 4
  2. 2025AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinkingcited by 4
  3. 2025TiCT: A Synthetically Pre-Trained Foundation Model for Time Series Classificationcited by 3
  4. 2025Learning richness modulates equality reasoning in neural networkscited by 2
  5. 2025Secret mixtures of experts inside your LLMcited by 2
  6. 2024Three Pathways to Neurosymbolic Reinforcement Learning with Interpretable Model and Policy Networkscited by 1
  7. 2025Optimization Problem Solving Can Transition to Evolutionary Agentic Workflowscited by 0
  8. 2026To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoningcited by 0
  9. 2026A mathematical theory of balancing relational generalization and memorizationcited by 0
  10. 2026Syntax vs. Semantics: How Transformers Learn Deep Dependenciescited by 0
  11. 2026Logical subspace in LLMscited by 0