mlsanityData to 14 Oct 2026

Meta-learning of shared linear representations beyond well-specified linear regression

arXiv 2501.18975 ↗First posted 31 Jan 2025cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Motivated by multi-task and meta-learning approaches, we consider the problem of learning structure shared by tasks or users, such as shared low-rank representations or clustered structures. While all previous works focus on well-specified linear regression, we consider more general convex objectives, where the structural low-rank and cluster assumptions are expressed on the optima of each function. We show that under mild assumptions such as textitHessian concentration and textitnoise concentration at the optimum, rank and clustered regularized estimators recover such structure, provided the number of samples per task and the number of tasks are large enough. We then study the problem of recovering the subspace in which all the solutions lie, in the setting where there is only a single sample per task: we show that in that case, the rank-constrained estimator can recover the subspace, but that the number of tasks needs to scale exponentially large with the dimension of the subspace. Finally, we provide a polynomial-time algorithm via nuclear norm constraints for learning a shared linear representation in the context of convex learning objectives.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 55 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

23 of 55 references.

References · 23

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  3. 2014Convex Optimization: Algorithms and Complexitycited by 575
  4. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  5. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  6. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  7. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  8. 2015The Benefit of Multitask Representation Learningcited by 195
  9. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  10. 2020Provable Meta-Learning of Linear Representationscited by 161
  11. 2020Lower Bounds and Optimal Algorithms for Personalized Federated Learningcited by 123
  12. 2019Provable Guarantees for Gradient-Based Meta-Learningcited by 67
11 more
  1. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 56
  2. 2019Learning-to-Learn Stochastic Gradient Descent with Biased Regularizationcited by 49
  3. 2022MAML and ANIL Provably Learn Representationscited by 30
  4. 2023Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networkscited by 16
  5. 2022Subspace Recovery from Heterogeneous Data with Non-isotropic Noisecited by 14
  6. 2015Convex Learning of Multiple Tasks and their Structurecited by 13
  7. 2023Interpretable Meta-Learning of Physical Systemscited by 11
  8. 2022Trace norm regularization for multi-task learning with scarce datacited by 10
  9. 2021Multitask Online Mirror Descentcited by 6
  10. 2023Nonparametric Linear Feature Learning in Regression Through Regularisationcited by 5
  11. 2021Concentration of Non-Isotropic Random Tensors with Applications to Learning and Empirical Risk Minimizationcited by 4

Built on by

Not cited yet.