mlsanityData to 14 Oct 2026

Learning to Extrapolate: A Transductive Approach

arXiv 2304.14329 ↗First posted 27 Apr 2023cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Machine learning systems, especially with overparameterized deep neural networks, can generalize to novel test instances drawn from the same distribution as the training data. However, they fare poorly when evaluated on out-of-support test points. In this work, we tackle the problem of developing machine learning systems that retain the power of overparameterized function approximators while enabling extrapolation to out-of-support test points when possible. This is accomplished by noting that under certain conditions, a "transductive" reparameterization can convert an out-of-support extrapolation problem into a problem of within-support combinatorial generalization. We propose a simple strategy based on bilinear embeddings to enable this type of combinatorial generalization, thereby addressing the out-of-support extrapolation problem under certain conditions. We instantiate a simple, practical algorithm applicable to various supervised learning and imitation learning tasks.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 58 of its 95 references that are in the map. First posted on arXiv 27 Apr 2023.

Builds on

58 of 95 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  9. 2015Trust Region Policy Optimizationcited by 2,580
  10. 2016Matching Networks for One Shot Learningcited by 2,255
  11. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  12. 2020Reformer: The Efficient Transformercited by 1,668
38 more
  1. 2017Deep Setscited by 1,421
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  3. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  4. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  6. 2016Group Equivariant Convolutional Networkscited by 1,165
  7. 2017Hindsight Experience Replaycited by 1,136
  8. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 746
  9. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
  10. 2015Neural Module Networkscited by 376
  11. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
  12. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 337
  13. 2021Vector Neurons: A General Framework for SO(3)-Equivariant Networkscited by 305
  14. 2019Causal Confusion in Imitation Learningcited by 288
  15. 2021Combinatorial optimization and reasoning with graph neural networkscited by 268
  16. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  17. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  18. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 209
  19. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 153
  20. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  21. 2020On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Locationcited by 142
  22. 2020Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?cited by 139
  23. 2021Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue Systemcited by 139
  24. 2019On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithmscited by 132
  25. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 126
  26. 2013Learning by Transductioncited by 121
  27. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 109
  28. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  29. 2021Abstraction and Analogy-Making in Artificial Intelligencecited by 103
  30. 2019Meta-Learning without Memorizationcited by 92
  31. 2019Measuring Compositionality in Representation Learningcited by 84
  32. 2022SO(2)-Equivariant Reinforcement Learningcited by 84
  33. 2021Automatic Symmetry Discovery with Lie Algebra Convolutional Networkcited by 83
  34. 2016Deep Feature Interpolation for Image Content Changescited by 80
  35. 2013Revisiting the Nystrom Method for Improved Large-Scale Machine Learningcited by 77
  36. 2022Theory of Graph Neural Networks: Representation and Learningcited by 61
  37. 2016One-shot Learning with Memory-Augmented Neural Networkscited by 60
  38. 2021Causal Matrix Completioncited by 45

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 5; 2026: 4.

19 papers.

Sort:

By area:

  1. 2023Compositional Generalization from First Principlescited by 64
  2. 2023Provable Compositional Generalization for Object-Centric Learningcited by 41
  3. 2024On Provable Length and Compositional Generalizationcited by 22
  4. 2024Accurate and Scalable Estimation of Epistemic Uncertainty for Graph Neural Networkscited by 8
  5. 2025Known Unknowns: Out-of-Distribution Property Prediction in Materials and Moleculescited by 7
  6. 2024A General Theory for Compositional Generalizationcited by 4
  7. 2025Towards Understanding Extrapolation: a Causal Lenscited by 4
  8. 2025Large Language Models for Extrapolative Modeling of Manufacturing Processescited by 4
  9. 2024Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learningcited by 2
  10. 2024Advancing Extrapolative Predictions of Material Properties through Learning to Learncited by 2
  11. 2024On the Use of Anchoring for Training Vision Modelscited by 1
  12. 2025Regularizing Extrapolation in Causal Inferencecited by 1
7 more
  1. 2026Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learningcited by 1
  2. 2023Accurate and Scalable Estimation of Epistemic Uncertainty for Graph Neural Networkscited by 0
  3. 2023PAGER: A Framework for Failure Analysis of Deep Regression Modelscited by 0
  4. 2025Learning Model Successorscited by 0
  5. 2026Learning to Extrapolate to New Tasks: A Relational Approach to Task Extrapolationcited by 0
  6. 2026Physics-Distilled Neural Network enabled by Large Language Models for Manufacturing Process-Property Predictive Modelingcited by 0
  7. 2026Distributional Extrapolation for Interactionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking
  2. 2024Beyond Finite Data: Towards Data-free Out-of-distribution Generalization via Extrapolation
  3. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
  4. 2025Machine Unlearning under Overparameterization
  5. 2025Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
  6. 2022Learning and Extrapolation of Robotic Skills using Task-Parameterized Equation Learner Networks
  7. 2024Toward generalizable learning of all (linear) first-order methods via memory augmented Transformers
  8. 2025Beyond Interpolation: Extrapolative Reasoning with Reinforcement Learning and Graph Neural Networks