# Active Multi-Task Representation Learning

arXiv: https://arxiv.org/abs/2202.00911 · HTML: https://mlsanity.com/paper/2202.00911/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 20 references that are in the map.
- First posted on arXiv 2 Feb 2022.

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [Few-Shot Learning via Learning the Representation, Provably](https://mlsanity.com/paper/2002.09434/) (2020)
- [On the Theory of Transfer Learning: The Importance of Task Diversity](https://mlsanity.com/paper/2006.11650/) (2020)
- [Provable Meta-Learning of Linear Representations](https://mlsanity.com/paper/2002.11684/) (2020)
- [MNIST-C: A Robustness Benchmark for Computer Vision](https://mlsanity.com/paper/1906.02337/) (2019)
- [How Fine-Tuning Allows for Effective Meta-Learning](https://mlsanity.com/paper/2105.02221/) (2021)
- [NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework](https://mlsanity.com/paper/2111.04130/) (2021)
- [Sample Efficient Linear Meta-Learning by Alternating Minimization](https://mlsanity.com/paper/2105.08306/) (2021)
- [Weighted Training for Cross-Task Learning](https://mlsanity.com/paper/2105.14095/) (2021)
- [Representation Learning Beyond Linear Prediction Functions](https://mlsanity.com/paper/2105.14989/) (2021)
- [A Theoretical Analysis of Fine-tuning with Linear Teachers](https://mlsanity.com/paper/2107.01641/) (2021)

## Built on by (most cited first, 14 of 14)

- [Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/2307.06887/) (2023)
- [Offline Multi-task Transfer RL with Representational Penalization](https://mlsanity.com/paper/2402.12570/) (2024)
- [Improved Active Multi-Task Representation Learning via Lasso](https://mlsanity.com/paper/2306.02556/) (2023)
- [Blessing of Class Diversity in Pre-training](https://mlsanity.com/paper/2209.03447/) (2022)
- [Active Representation Learning for General Task Space with Applications in Robotics](https://mlsanity.com/paper/2306.08942/) (2023)
- [Guarantees for Nonlinear Representation Learning: Non-identical Covariates, Dependent Data, Fewer Samples](https://mlsanity.com/paper/2410.11227/) (2024)
- [The Power of Active Multi-Task Learning in Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2405.11226/) (2024)
- [Relational Learning in Pre-Trained Models: A Theory from Hypergraph Recovery Perspective](https://mlsanity.com/paper/2406.11249/) (2024)
- [Bridging Lifelong and Multi-Task Representation Learning via Algorithm and Complexity Measure](https://mlsanity.com/paper/2511.01847/) (2025)
- [Beyond Centralization: Provable Communication Efficient Decentralized Multi-Task Learning](https://mlsanity.com/paper/2512.22675/) (2025)
- [Diffusion-based Decentralized Federated Multi-Task Representation Learning](https://mlsanity.com/paper/2512.23161/) (2025)
- [RepUCB: Representation Learning-Based UCB for Heterogeneous Multi-Task Linear Bandits](https://mlsanity.com/paper/2604.00531/) (2026)
- [Multi-Task Representation Learning for Conservative Linear Bandits](https://mlsanity.com/paper/2605.12176/) (2026)
- [How Shared Representations Become Specialized: Formation, Maintenance, and Latent Susceptibility](https://mlsanity.com/paper/2610.15830/) (2026)

## Abstract (arXiv)

To leverage the power of big data from source tasks and overcome the scarcity of the target task samples, representation learning based on multi-task pretraining has become a standard approach in many applications. However, up until now, choosing which source tasks to include in the multi-task learning has been more art than science. In this paper, we give the first formal study on resource task sampling by leveraging the techniques from active learning. We propose an algorithm that iteratively estimates the relevance of each source task to the target task and samples from each source task based on the estimated relevance. Theoretically, we show that for the linear representation class, to achieve the same error rate, our algorithm can save up to a textitnumber of source tasks factor in the source task sample complexity, compared with the naive uniform sampling from all source tasks. We also provide experiments on real-world computer vision datasets to illustrate the effectiveness of our proposed method on both linear and convolutional neural network representation classes. We believe our paper serves as an important initial step to bring techniques from active learning to representation learning.