# Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm

arXiv: https://arxiv.org/abs/1710.11622 · HTML: https://mlsanity.com/paper/1710.11622/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2017.

## Built on by (most cited first, 30 of 65)

- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [Offline Meta-Reinforcement Learning with Advantage Weighting](https://mlsanity.com/paper/2008.06043/) (2020)
- [Multi-Objective Loss Balancing for Physics-Informed Deep Learning](https://mlsanity.com/paper/2110.09813/) (2021)
- [Advances and Challenges in Meta-Learning: A Technical Review](https://mlsanity.com/paper/2307.04722/) (2023)
- [Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learning](https://mlsanity.com/paper/2206.03996/) (2022)
- [C-Mixup: Improving Generalization in Regression](https://mlsanity.com/paper/2210.05775/) (2022)
- [Evolving Reinforcement Learning Algorithms](https://mlsanity.com/paper/2101.03958/) (2021)
- [Pin the Memory: Learning to Generalize Semantic Segmentation](https://mlsanity.com/paper/2204.03609/) (2022)
- [Meta-Learning with Fewer Tasks through Task Interpolation](https://mlsanity.com/paper/2106.02695/) (2021)
- [A Modern Self-Referential Weight Matrix That Learns to Modify Itself](https://mlsanity.com/paper/2202.05780/) (2022)
- [A novel meta-learning initialization method for physics-informed neural networks](https://mlsanity.com/paper/2107.10991/) (2021)
- [Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights](https://mlsanity.com/paper/2603.12228/) (2026)
- [Meta-learning approaches for few-shot learning: A survey of recent advances](https://mlsanity.com/paper/2303.07502/) (2023)
- [How to Train Your MAML to Excel in Few-Shot Classification](https://mlsanity.com/paper/2106.16245/) (2021)
- [RestNet: Boosting Cross-Domain Few-Shot Segmentation with Residual Transformation Network](https://mlsanity.com/paper/2308.13469/) (2023)
- [Few-Shot Learning with a Strong Teacher](https://mlsanity.com/paper/2107.00197/) (2021)
- [CoMPS: Continual Meta Policy Search](https://mlsanity.com/paper/2112.04467/) (2021)
- [Conditional Meta-Learning of Linear Representations](https://mlsanity.com/paper/2103.16277/) (2021)
- [Learning to Rectify for Robust Learning with Noisy Labels](https://mlsanity.com/paper/2111.04239/) (2021)
- [When Meta-Learning Meets Online and Continual Learning: A Survey](https://mlsanity.com/paper/2311.05241/) (2023)
- [Meta-learning with negative learning rates](https://mlsanity.com/paper/2102.00940/) (2021)
- [Meta-Learning based Degradation Representation for Blind Super-Resolution](https://mlsanity.com/paper/2207.13963/) (2022)
- [Meta-learning Spiking Neural Networks with Surrogate Gradient Descent](https://mlsanity.com/paper/2201.10777/) (2022)
- [Test-Time Adaptation for Point Cloud Upsampling Using Meta-Learning](https://mlsanity.com/paper/2308.16484/) (2023)
- [Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution](https://mlsanity.com/paper/2102.11503/) (2021)
- [Provable Generalization of Overparameterized Meta-learning Trained with SGD](https://mlsanity.com/paper/2206.09136/) (2022)
- [Self-Attention Meta-Learner for Continual Learning](https://mlsanity.com/paper/2101.12136/) (2021)
- [Learning and Fast Adaptation for Grid Emergency Control via Deep Meta Reinforcement Learning](https://mlsanity.com/paper/2101.05317/) (2021)
- [Stateless Neural Meta-Learning using Second-Order Gradients](https://mlsanity.com/paper/2104.10527/) (2021)
- [Metalearning Continual Learning Algorithms](https://mlsanity.com/paper/2312.00276/) (2023)

## Abstract (arXiv)

Learning to learn is a powerful paradigm for enabling models to learn from data more effectively and efficiently. A popular approach to meta-learning is to train a recurrent model to read in a training dataset as input and output the parameters of a learned model, or output predictions for new test inputs. Alternatively, a more recent approach to meta-learning aims to acquire deep representations that can be effectively fine-tuned, via standard gradient descent, to new tasks. In this paper, we consider the meta-learning problem from the perspective of universality, formalizing the notion of learning algorithm approximation and comparing the expressive power of the aforementioned recurrent models to the more recent approaches that embed gradient descent into the meta-learner. In particular, we seek to answer the following question: does deep representation combined with standard gradient descent have sufficient capacity to approximate any learning algorithm? We find that this is indeed true, and further find, in our experiments, that gradient-based meta-learning consistently leads to learning strategies that generalize more widely compared to those represented by recurrent models.