# Learning from Similar Linear Representations: Adaptivity, Minimaxity, and Robustness

arXiv: https://arxiv.org/abs/2303.17765 · HTML: https://mlsanity.com/paper/2303.17765/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 9 references that are in the map.
- First posted on arXiv 31 Mar 2023.

## Builds on (most cited first, 6 of 6)

- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [Few-Shot Learning via Learning the Representation, Provably](https://mlsanity.com/paper/2002.09434/) (2020)
- [The Benefit of Multitask Representation Learning](https://mlsanity.com/paper/1505.06279/) (2015)
- [Provable Meta-Learning of Linear Representations](https://mlsanity.com/paper/2002.11684/) (2020)
- [Adaptive and Robust Multi-Task Learning](https://mlsanity.com/paper/2202.05250/) (2022)
- [Subspace Recovery from Heterogeneous Data with Non-isotropic Noise](https://mlsanity.com/paper/2210.13497/) (2022)

## Built on by (most cited first, 28 of 28)

- [TransFusion: Covariate-Shift Robust Transfer Learning for High-Dimensional Regression](https://mlsanity.com/paper/2404.01153/) (2024)
- [Nonlinear Meta-Learning Can Guarantee Faster Rates](https://mlsanity.com/paper/2307.10870/) (2023)
- [Smoothness Adaptive Hypothesis Transfer Learning](https://mlsanity.com/paper/2402.14966/) (2024)
- [Optimal Multitask Linear Regression and Contextual Bandits under Sparse Heterogeneity](https://mlsanity.com/paper/2306.06291/) (2023)
- [Multi-Task Learning with Summary Statistics](https://mlsanity.com/paper/2307.02388/) (2023)
- [Towards the Theory of Unsupervised Federated Learning: Non-asymptotic Analysis of Federated EM Algorithms](https://mlsanity.com/paper/2310.15330/) (2023)
- [Representation Retrieval Learning for Heterogeneous Data Integration](https://mlsanity.com/paper/2503.09494/) (2025)
- [Representational Transfer Learning for Matrix Completion](https://mlsanity.com/paper/2412.06233/) (2024)
- [LEARNER: A Transfer Learning Method for Low-Rank Matrix Estimation](https://mlsanity.com/paper/2412.20605/) (2024)
- [Multi-dimensional domain generalization with low-rank structures](https://mlsanity.com/paper/2309.09555/) (2023)
- [Few-shot Multi-Task Learning of Linear Invariant Features with Meta Subspace Pursuit](https://mlsanity.com/paper/2409.02708/) (2024)
- [Multi-Task Dynamic Pricing in Credit Market with Contextual Information](https://mlsanity.com/paper/2410.14839/) (2024)
- [Late Fusion Multi-task Learning for Semiparametric Inference with Nuisance Parameters](https://mlsanity.com/paper/2507.07941/) (2025)
- [Learning with Shared Representations: Statistical Rates and Efficient Algorithms](https://mlsanity.com/paper/2409.04919/) (2024)
- [Multi-task Learning for Heterogeneous Multi-source Block-Wise Missing Data](https://mlsanity.com/paper/2505.24413/) (2025)
- [Transfer Learning for High-dimensional Reduced Rank Time Series Models](https://mlsanity.com/paper/2504.15691/) (2025)
- [GeoERM: Geometry-Aware Multi-Task Representation Learning on Riemannian Manifolds](https://mlsanity.com/paper/2505.02972/) (2025)
- [Joint estimation of smooth graph signals from partial linear measurements](https://mlsanity.com/paper/2505.23240/) (2025)
- [Robust and Adaptive Spectral Method for Representation Multi-Task Learning with Contamination](https://mlsanity.com/paper/2509.06575/) (2025)
- [Representation-Aware Distributionally Robust Optimization: A Knowledge Transfer Framework](https://mlsanity.com/paper/2509.09371/) (2025)
- [On the Power of Source Screening for Learning Shared Feature Extractors](https://mlsanity.com/paper/2602.16125/) (2026)
- [Model-agnostic information transfer and fusion for classification with label noise](https://mlsanity.com/paper/2604.25845/) (2026)
- [Near-optimal and Efficient First-Order Algorithm for Multi-Task Learning with Shared Linear Representation](https://mlsanity.com/paper/2605.00473/) (2026)
- [Sample Complexity of Transfer Learning: An Optimal Transport Approach](https://mlsanity.com/paper/2605.20545/) (2026)
- [Deep Multitask Learning for Mixed-Type Outcomes with Shared Sparsity](https://mlsanity.com/paper/2607.00995/) (2026)
- [Contaminated Multi-task Learning with Heterogeneity: Fundamental Limits and Optimal Algorithms](https://mlsanity.com/paper/2607.02681/) (2026)
- [Capacity and Redundancy Trade-offs in Multi-Task Learning](https://mlsanity.com/paper/2607.16554/) (2026)
- [Transfer Learning in Nonparametric Regression with Deep ReLU Networks](https://mlsanity.com/paper/2608.20255/) (2026)

## Abstract (arXiv)

Representation multi-task learning (MTL) has achieved tremendous success in practice. However, the theoretical understanding of these methods is still lacking. Most existing theoretical works focus on cases where all tasks share the same representation, and claim that MTL almost always improves performance. Nevertheless, as the number of tasks grows, assuming all tasks share the same representation is unrealistic. Furthermore, empirical findings often indicate that a shared representation does not necessarily improve single-task learning performance. In this paper, we aim to understand how to learn from tasks with textitsimilar but not exactly the same linear representations, while dealing with outlier tasks. Assuming a known intrinsic dimension, we propose a penalized empirical risk minimization method and a spectral method that are textitadaptive to the similarity structure and textitrobust to outlier tasks. Both algorithms outperform single-task learning when representations across tasks are sufficiently similar and the proportion of outlier tasks is small. Moreover, they always perform at least as well as single-task learning, even when the representations are dissimilar. We provide information-theoretic lower bounds to demonstrate that both methods are nearly textitminimax optimal in a large regime, with the spectral method being optimal in the absence of outlier tasks. Additionally, we introduce a thresholding algorithm to adapt to an unknown intrinsic dimension. We conduct extensive numerical experiments to validate our theoretical findings.