# Functional Indirection Neural Estimator for Better Out-of-distribution Generalization

arXiv: https://arxiv.org/abs/2210.12739 · HTML: https://mlsanity.com/paper/2210.12739/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 23 Oct 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [RAVEN: A Dataset for Relational and Analogical Visual rEasoNing](https://mlsanity.com/paper/1903.02741/) (2019)
- [The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers](https://mlsanity.com/paper/2108.12284/) (2021)
- [Learning Independent Causal Mechanisms](https://mlsanity.com/paper/1712.00961/) (2017)
- [Abstraction and Analogy-Making in Artificial Intelligence](https://mlsanity.com/paper/2102.10717/) (2021)
- [Learning to Remember Rare Events](https://mlsanity.com/paper/1703.03129/) (2017)
- [Emergent Symbols through Binding in External Memory](https://mlsanity.com/paper/2012.14601/) (2020)
- [Metalearned Neural Memory](https://mlsanity.com/paper/1907.09720/) (2019)
- [Learning Representations that Support Extrapolation](https://mlsanity.com/paper/2007.05059/) (2020)
- [An Explicitly Relational Neural Network Architecture](https://mlsanity.com/paper/1905.10307/) (2019)
- [Dynamic Inference with Neural Interpreters](https://mlsanity.com/paper/2110.06399/) (2021)
- [Neural Stored-program Memory](https://mlsanity.com/paper/1906.08862/) (2019)

## Abstract (arXiv)

The capacity to achieve out-of-distribution (OOD) generalization is a hallmark of human intelligence and yet remains out of reach for machines. This remarkable capability has been attributed to our abilities to make conceptual abstraction and analogy, and to a mechanism known as indirection, which binds two representations and uses one representation to refer to the other. Inspired by these mechanisms, we hypothesize that OOD generalization may be achieved by performing analogy-making and indirection in the functional space instead of the data space as in current methods. To realize this, we design FINE (Functional Indirection Neural Estimator), a neural framework that learns to compose functions that map data input to output on-the-fly. FINE consists of a backbone network and a trainable semantic memory of basis weight matrices. Upon seeing a new input-output data pair, FINE dynamically constructs the backbone weights by mixing the basis weights. The mixing coefficients are indirectly computed through querying a separate corresponding semantic memory using the data pair. We demonstrate empirically that FINE can strongly improve out-of-distribution generalization on IQ tasks that involve geometric transformations. In particular, we train FINE and competing models on IQ tasks using images from the MNIST, Omniglot and CIFAR100 datasets and test on tasks with unseen image classes from one or different datasets and unseen transformation rules. FINE not only achieves the best performance on all tasks but also is able to adapt to small-scale data scenarios.