# Learning Representations that Support Extrapolation

arXiv: https://arxiv.org/abs/2007.05059 · HTML: https://mlsanity.com/paper/2007.05059/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2020.

## Built on by (most cited first, 30 of 36)

- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [Abstraction and Analogy-Making in Artificial Intelligence](https://mlsanity.com/paper/2102.10717/) (2021)
- [Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation](https://mlsanity.com/paper/2307.02598/) (2023)
- [Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matrices](https://mlsanity.com/paper/2201.12382/) (2022)
- [A Review of Emerging Research Directions in Abstract Visual Reasoning](https://mlsanity.com/paper/2202.10284/) (2022)
- [Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus](https://mlsanity.com/paper/2403.11793/) (2024)
- [MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning](https://mlsanity.com/paper/2404.13591/) (2024)
- [MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Models](https://mlsanity.com/paper/2502.00698/) (2025)
- [On Neural Architecture Inductive Biases for Relational Tasks](https://mlsanity.com/paper/2206.05056/) (2022)
- [When can transformers reason with abstract symbols?](https://mlsanity.com/paper/2310.09753/) (2023)
- [Systematic Visual Reasoning through Object-Centric Relational Abstraction](https://mlsanity.com/paper/2306.02500/) (2023)
- [Learning to reason over visual objects](https://mlsanity.com/paper/2303.02260/) (2023)
- [Slot Abstractors: Toward Scalable Abstract Visual Reasoning](https://mlsanity.com/paper/2403.03458/) (2024)
- [Zero-shot visual reasoning through probabilistic analogical mapping](https://mlsanity.com/paper/2209.15087/) (2022)
- [Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study](https://mlsanity.com/paper/2209.07736/) (2022)
- [ARCLE: The Abstraction and Reasoning Corpus Learning Environment for Reinforcement Learning](https://mlsanity.com/paper/2407.20806/) (2024)
- [Bongard-RWR+: Real-World Representations of Fine-Grained Concepts in Bongard Problems](https://mlsanity.com/paper/2508.12026/) (2025)
- [Imagine the Unseen World: A Benchmark for Systematic Generalization in Visual World Models](https://mlsanity.com/paper/2311.09064/) (2023)
- [COLUMBUS: Evaluating COgnitive Lateral Understanding through Multiple-choice reBUSes](https://mlsanity.com/paper/2409.04053/) (2024)
- [Modelling the development of counting with memory-augmented neural networks](https://mlsanity.com/paper/2105.10577/) (2021)
- [Extrapolation Frameworks in Cognitive Psychology Suitable for Study of Image Classification Models](https://mlsanity.com/paper/2112.03411/) (2021)
- [A-I-RAVEN and I-RAVEN-Mesh: Two New Benchmarks for Abstract Visual Reasoning](https://mlsanity.com/paper/2406.11061/) (2024)
- [A Self-Supervised Framework for Function Learning and Extrapolation](https://mlsanity.com/paper/2106.07369/) (2021)
- [Learning Hierarchical Relational Representations through Relational Convolutions](https://mlsanity.com/paper/2310.03240/) (2023)
- [To what extent should we trust AI models when they extrapolate?](https://mlsanity.com/paper/2201.11260/) (2022)
- [Deep Neural Networks Tend To Extrapolate Predictably](https://mlsanity.com/paper/2310.00873/) (2023)
- [Should Machine Learning Models Report to Us When They Are Clueless?](https://mlsanity.com/paper/2203.12131/) (2022)
- [Functional Indirection Neural Estimator for Better Out-of-distribution Generalization](https://mlsanity.com/paper/2210.12739/) (2022)
- [Determinantal Point Process Attention Over Grid Cell Code Supports Out of Distribution Generalization](https://mlsanity.com/paper/2305.18417/) (2023)
- [The Extrapolation Power of Implicit Models](https://mlsanity.com/paper/2407.14430/) (2024)

## Abstract (arXiv)

Extrapolation -- the ability to make inferences that go beyond the scope of one's experiences -- is a hallmark of human intelligence. By contrast, the generalization exhibited by contemporary neural network algorithms is largely limited to interpolation between data points in their training corpora. In this paper, we consider the challenge of learning representations that support extrapolation. We introduce a novel visual analogy benchmark that allows the graded evaluation of extrapolation as a function of distance from the convex domain defined by the training data. We also introduce a simple technique, temporal context normalization, that encourages representations that emphasize the relations between objects. We find that this technique enables a significant improvement in the ability to extrapolate, considerably outperforming a number of competitive techniques.