# Abstractors and relational cross-attention: An inductive bias for explicit relational reasoning in Transformers

arXiv: https://arxiv.org/abs/2304.00195 · HTML: https://mlsanity.com/paper/2304.00195/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 1 Apr 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [Measuring abstract reasoning in neural networks](https://mlsanity.com/paper/1807.04225/) (2018)
- [Neural Episodic Control](https://mlsanity.com/paper/1703.01988/) (2017)
- [Emergent Symbols through Binding in External Memory](https://mlsanity.com/paper/2012.14601/) (2020)
- [An Explicitly Relational Neural Network Architecture](https://mlsanity.com/paper/1905.10307/) (2019)
- [On Neural Architecture Inductive Biases for Relational Tasks](https://mlsanity.com/paper/2206.05056/) (2022)
- [Learning to reason over visual objects](https://mlsanity.com/paper/2303.02260/) (2023)

## Built on by (most cited first, 12 of 12)

- [Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models](https://mlsanity.com/paper/2502.20332/) (2025)
- [The Relational Bottleneck as an Inductive Bias for Efficient Abstraction](https://mlsanity.com/paper/2309.06629/) (2023)
- [Approximation of relation functions and attention mechanisms](https://mlsanity.com/paper/2402.08856/) (2024)
- [Disentangling and Integrating Relational and Sensory Information in Transformer Architectures](https://mlsanity.com/paper/2405.16727/) (2024)
- [Learning Hierarchical Relational Representations through Relational Convolutions](https://mlsanity.com/paper/2310.03240/) (2023)
- [Language as a Wave Phenomenon: Semantic Phase Locking and Interference in Neural Networks](https://mlsanity.com/paper/2512.01208/) (2025)
- [SceneNAT: Masked Generative Modeling for Language-Guided Indoor Scene Synthesis](https://mlsanity.com/paper/2601.07218/) (2026)
- [Systematic Abductive Reasoning via Diverse Relation Representations in Vector-symbolic Architecture](https://mlsanity.com/paper/2501.11896/) (2025)
- [Convolutional Neural Networks Can (Meta-)Learn the Same-Different Relation](https://mlsanity.com/paper/2503.23212/) (2025)
- [A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents](https://mlsanity.com/paper/2510.22052/) (2025)
- [Shortcut Solutions Learned by Transformers Impair Continual Compositional Reasoning](https://mlsanity.com/paper/2605.05495/) (2026)
- [Relational Attention for Data-Efficient Language Modeling](https://mlsanity.com/paper/2609.20530/) (2026)

## Abstract (arXiv)

An extension of Transformers is proposed that enables explicit relational reasoning through a novel module called the Abstractor. At the core of the Abstractor is a variant of attention called relational cross-attention. The approach is motivated by an architectural inductive bias for relational learning that disentangles relational information from object-level features. This enables explicit relational reasoning, supporting abstraction and generalization from limited data. The Abstractor is first evaluated on simple discriminative relational tasks and compared to existing relational architectures. Next, the Abstractor is evaluated on purely relational sequence-to-sequence tasks, where dramatic improvements are seen in sample efficiency compared to standard Transformers. Finally, Abstractors are evaluated on a collection of tasks based on mathematical problem solving, where consistent improvements in performance and sample efficiency are observed.