# The Relational Bottleneck as an Inductive Bias for Efficient Abstraction

arXiv: https://arxiv.org/abs/2309.06629 · HTML: https://mlsanity.com/paper/2309.06629/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 138 references that are in the map.
- First posted on arXiv 12 Sep 2023.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [Neural Episodic Control](https://mlsanity.com/paper/1703.01988/) (2017)
- [Inferring and Executing Programs for Visual Reasoning](https://mlsanity.com/paper/1705.03633/) (2017)
- [Learning Neural Causal Models from Unknown Interventions](https://mlsanity.com/paper/1910.01075/) (2019)
- [Learning to Compose Neural Networks for Question Answering](https://mlsanity.com/paper/1601.01705/) (2016)
- [Neural Production Systems: Learning Rule-Governed Visual Dynamics](https://mlsanity.com/paper/2103.01937/) (2021)
- [Learning Compositional Rules via Neural Program Synthesis](https://mlsanity.com/paper/2003.05562/) (2020)
- [A comparison of Vector Symbolic Architectures](https://mlsanity.com/paper/2001.11797/) (2020)
- [Compositional generalization in a deep seq2seq model by separating syntax and semantics](https://mlsanity.com/paper/1904.09708/) (2019)
- [Emergent Symbols through Binding in External Memory](https://mlsanity.com/paper/2012.14601/) (2020)
- [Discrete-Valued Neural Communication](https://mlsanity.com/paper/2107.02367/) (2021)
- [Understanding Human Intelligence through Human Limitations](https://mlsanity.com/paper/2009.14050/) (2020)
- [On Neural Architecture Inductive Biases for Relational Tasks](https://mlsanity.com/paper/2206.05056/) (2022)
- [One Model for the Learning of Language](https://mlsanity.com/paper/1711.06301/) (2017)
- [Systematic Visual Reasoning through Object-Centric Relational Abstraction](https://mlsanity.com/paper/2306.02500/) (2023)
- [Learning to reason over visual objects](https://mlsanity.com/paper/2303.02260/) (2023)
- [Probabilistic Analogical Mapping with Semantic Relation Networks](https://mlsanity.com/paper/2103.16704/) (2021)

## Built on by (most cited first, 27 of 27)

- [Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models](https://mlsanity.com/paper/2502.20332/) (2025)
- [What is the Visual Cognition Gap between Humans and Multimodal LLMs?](https://mlsanity.com/paper/2406.10424/) (2024)
- [When can transformers reason with abstract symbols?](https://mlsanity.com/paper/2310.09753/) (2023)
- [Evidence from counterfactual tasks supports emergent analogical reasoning in large language models](https://mlsanity.com/paper/2404.13070/) (2024)
- [MLPs Learn In-Context on Regression and Classification Tasks](https://mlsanity.com/paper/2405.15618/) (2024)
- [Visual symbolic mechanisms: Emergent symbol processing in vision language models](https://mlsanity.com/paper/2506.15871/) (2025)
- [Slot Abstractors: Toward Scalable Abstract Visual Reasoning](https://mlsanity.com/paper/2403.03458/) (2024)
- [From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks](https://mlsanity.com/paper/2405.15164/) (2024)
- [Whither symbols in the era of advanced neural networks?](https://mlsanity.com/paper/2508.05776/) (2025)
- [Disentangling and Integrating Relational and Sensory Information in Transformer Architectures](https://mlsanity.com/paper/2405.16727/) (2024)
- [Deep Neural Networks Can Learn Generalizable Same-Different Visual Relations](https://mlsanity.com/paper/2310.09612/) (2023)
- [Human-Like Geometric Abstraction in Large Pre-trained Neural Networks](https://mlsanity.com/paper/2402.04203/) (2024)
- [Unraveling the geometry of visual relational reasoning](https://mlsanity.com/paper/2502.17382/) (2025)
- [Learning Hierarchical Relational Representations through Relational Convolutions](https://mlsanity.com/paper/2310.03240/) (2023)
- [Learning richness modulates equality reasoning in neural networks](https://mlsanity.com/paper/2503.09781/) (2025)
- [Language as a Wave Phenomenon: Semantic Phase Locking and Interference in Neural Networks](https://mlsanity.com/paper/2512.01208/) (2025)
- [Visual Reasoning in Object-Centric Deep Neural Networks: A Comparative Cognition Approach](https://mlsanity.com/paper/2402.12675/) (2024)
- [A Relational Inductive Bias for Dimensional Abstraction in Neural Networks](https://mlsanity.com/paper/2402.18426/) (2024)
- [LARS-VSA: A Vector Symbolic Architecture For Learning with Abstract Rules](https://mlsanity.com/paper/2405.14436/) (2024)
- [Systematic Abductive Reasoning via Diverse Relation Representations in Vector-symbolic Architecture](https://mlsanity.com/paper/2501.11896/) (2025)
- [Convolutional Neural Networks Can (Meta-)Learn the Same-Different Relation](https://mlsanity.com/paper/2503.23212/) (2025)
- [A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents](https://mlsanity.com/paper/2510.22052/) (2025)
- [A mathematical theory of balancing relational generalization and memorization](https://mlsanity.com/paper/2605.22972/) (2026)
- [Relational Attention for Data-Efficient Language Modeling](https://mlsanity.com/paper/2609.20530/) (2026)
- [Exactness at Inference: A Representational Criterion for Out-of-Distribution Generalization](https://mlsanity.com/paper/2609.24942/) (2026)
- [Neural Structural Reasoner: A Brain-inspired Architecture for Reasoning over Structured Knowledge](https://mlsanity.com/paper/2609.36620/) (2026)
- [Matching Object or Relation? Tracing Abstract Reasoning Inside VLMs](https://mlsanity.com/paper/2610.07646/) (2026)

## Abstract (arXiv)

A central challenge for cognitive science is to explain how abstract concepts are acquired from limited experience. This has often been framed in terms of a dichotomy between connectionist and symbolic cognitive models. Here, we highlight a recently emerging line of work that suggests a novel reconciliation of these approaches, by exploiting an inductive bias that we term the relational bottleneck. In that approach, neural networks are constrained via their architecture to focus on relations between perceptual inputs, rather than the attributes of individual inputs. We review a family of models that employ this approach to induce abstractions in a data-efficient manner, emphasizing their potential as candidate models for the acquisition of abstract concepts in the human mind and brain.