# Few-Shot Learning via Learning the Representation, Provably

arXiv: https://arxiv.org/abs/2002.09434 · HTML: https://mlsanity.com/paper/2002.09434/

## Key facts

- Cited by 214 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2020.

## Built on by (most cited first, 30 of 214)

- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection](https://mlsanity.com/paper/2306.04637/) (2023)
- [Transformers as Algorithms: Generalization and Stability in In-context Learning](https://mlsanity.com/paper/2301.07067/) (2023)
- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [A Kernel-Based View of Language Model Fine-Tuning](https://mlsanity.com/paper/2210.05643/) (2022)
- [The Expressive Power of Low-Rank Adaptation](https://mlsanity.com/paper/2310.17513/) (2023)
- [On the Role of Neural Collapse in Transfer Learning](https://mlsanity.com/paper/2112.15121/) (2021)
- [Universal Prompt Tuning for Graph Neural Networks](https://mlsanity.com/paper/2209.15240/) (2022)
- [Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models](https://mlsanity.com/paper/2210.14199/) (2022)
- [Variational Model Inversion Attacks](https://mlsanity.com/paper/2201.10787/) (2022)
- [RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation](https://mlsanity.com/paper/2405.00254/) (2024)
- [LoRA ensembles for large language model fine-tuning](https://mlsanity.com/paper/2310.00035/) (2023)
- [FedAvg with Fine Tuning: Local Updates Lead to Representation Learning](https://mlsanity.com/paper/2205.13692/) (2022)
- [Model Zoo: A Growing "Brain" That Learns Continually](https://mlsanity.com/paper/2106.03027/) (2021)
- [The Power of Contrast for Feature Learning: A Theoretical Analysis](https://mlsanity.com/paper/2110.02473/) (2021)
- [Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspective](https://mlsanity.com/paper/2308.13985/) (2023)
- [Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation](https://mlsanity.com/paper/2106.09017/) (2021)
- [Near-optimal Representation Learning for Linear Bandits and Linear RL](https://mlsanity.com/paper/2102.04132/) (2021)
- [Meta-Learning with Fewer Tasks through Task Interpolation](https://mlsanity.com/paper/2106.02695/) (2021)
- [Near-Optimal Linear Regression under Distribution Shift](https://mlsanity.com/paper/2106.12108/) (2021)
- [How Fine-Tuning Allows for Effective Meta-Learning](https://mlsanity.com/paper/2105.02221/) (2021)
- [Provable Benefits of Representational Transfer in Reinforcement Learning](https://mlsanity.com/paper/2205.14571/) (2022)
- [Invariant and Transportable Representations for Anti-Causal Domain Shifts](https://mlsanity.com/paper/2207.01603/) (2022)
- [Trainable Projected Gradient Method for Robust Fine-tuning](https://mlsanity.com/paper/2303.10720/) (2023)
- [Transformers are Minimax Optimal Nonparametric In-Context Learners](https://mlsanity.com/paper/2408.12186/) (2024)
- [Adversarial Training Helps Transfer Learning via Better Representations](https://mlsanity.com/paper/2106.10189/) (2021)
- [The Ideal Continual Learner: An Agent That Never Forgets](https://mlsanity.com/paper/2305.00316/) (2023)
- [Adaptive and Robust Multi-Task Learning](https://mlsanity.com/paper/2202.05250/) (2022)
- [Combining Observational and Randomized Data for Estimating Heterogeneous Treatment Effects](https://mlsanity.com/paper/2202.12891/) (2022)

## Abstract (arXiv)

This paper studies few-shot learning via representation learning, where one uses T source tasks with n1 data per task to learn a representation in order to reduce the sample complexity of a target task for which there is only n2 (ll n1) data. Specifically, we focus on the setting where there exists a good common representation between source and target, and our goal is to understand how much of a sample size reduction is possible. First, we study the setting where this common representation is low-dimensional and provide a fast rate of Oleft(fracCleft(Phiright)n1T + frackn2right); here, Phi is the representation function class, Cleft(Phiright) is its complexity measure, and k is the dimension of the representation. When specialized to linear representation functions, this rate becomes Oleft(fracdkn1T + frackn2right) where d (gg k) is the ambient input dimension, which is a substantial improvement over the rate without using representation learning, i.e. over the rate of Oleft(fracdn2right). This result bypasses the Omega(frac1T) barrier under the i.i.d. task assumption, and can capture the desired property that all n1T samples from source tasks can be pooled together for representation learning. Next, we consider the setting where the common representation may be high-dimensional but is capacity-constrained (say in norm); here, we again demonstrate the advantage of representation learning in both high-dimensional linear regression and neural network learning. Our results demonstrate representation learning can fully utilize all n1T samples from source tasks.