# Learning to Extrapolate: A Transductive Approach

arXiv: https://arxiv.org/abs/2304.14329 · HTML: https://mlsanity.com/paper/2304.14329/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 95 references that are in the map.
- First posted on arXiv 27 Apr 2023.

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [Certifying Some Distributional Robustness with Principled Adversarial Training](https://mlsanity.com/paper/1710.10571/) (2017)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019)
- [Combinatorial optimization and reasoning with graph neural networks](https://mlsanity.com/paper/2102.09544/) (2021)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation](https://mlsanity.com/paper/2112.05124/) (2021)

## Built on by (most cited first, 19 of 19)

- [Compositional Generalization from First Principles](https://mlsanity.com/paper/2307.05596/) (2023)
- [Provable Compositional Generalization for Object-Centric Learning](https://mlsanity.com/paper/2310.05327/) (2023)
- [On Provable Length and Compositional Generalization](https://mlsanity.com/paper/2402.04875/) (2024)
- [Accurate and Scalable Estimation of Epistemic Uncertainty for Graph Neural Networks](https://mlsanity.com/paper/2401.03350/) (2024)
- [Known Unknowns: Out-of-Distribution Property Prediction in Materials and Molecules](https://mlsanity.com/paper/2502.05970/) (2025)
- [A General Theory for Compositional Generalization](https://mlsanity.com/paper/2405.11743/) (2024)
- [Towards Understanding Extrapolation: a Causal Lens](https://mlsanity.com/paper/2501.09163/) (2025)
- [Large Language Models for Extrapolative Modeling of Manufacturing Processes](https://mlsanity.com/paper/2502.12185/) (2025)
- [Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning](https://mlsanity.com/paper/2404.04682/) (2024)
- [Advancing Extrapolative Predictions of Material Properties through Learning to Learn](https://mlsanity.com/paper/2404.08657/) (2024)
- [On the Use of Anchoring for Training Vision Models](https://mlsanity.com/paper/2406.00529/) (2024)
- [Regularizing Extrapolation in Causal Inference](https://mlsanity.com/paper/2509.17180/) (2025)
- [Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2605.20609/) (2026)
- [Accurate and Scalable Estimation of Epistemic Uncertainty for Graph Neural Networks](https://mlsanity.com/paper/2309.10976/) (2023)
- [PAGER: A Framework for Failure Analysis of Deep Regression Models](https://mlsanity.com/paper/2309.10977/) (2023)
- [Learning Model Successors](https://mlsanity.com/paper/2502.00197/) (2025)
- [Learning to Extrapolate to New Tasks: A Relational Approach to Task Extrapolation](https://mlsanity.com/paper/2605.30132/) (2026)
- [Physics-Distilled Neural Network enabled by Large Language Models for Manufacturing Process-Property Predictive Modeling](https://mlsanity.com/paper/2606.11605/) (2026)
- [Distributional Extrapolation for Interactions](https://mlsanity.com/paper/2608.19849/) (2026)

## Abstract (arXiv)

Machine learning systems, especially with overparameterized deep neural networks, can generalize to novel test instances drawn from the same distribution as the training data. However, they fare poorly when evaluated on out-of-support test points. In this work, we tackle the problem of developing machine learning systems that retain the power of overparameterized function approximators while enabling extrapolation to out-of-support test points when possible. This is accomplished by noting that under certain conditions, a "transductive" reparameterization can convert an out-of-support extrapolation problem into a problem of within-support combinatorial generalization. We propose a simple strategy based on bilinear embeddings to enable this type of combinatorial generalization, thereby addressing the out-of-support extrapolation problem under certain conditions. We instantiate a simple, practical algorithm applicable to various supervised learning and imitation learning tasks.