mlsanityData to 7 Oct 2026

On the Power of Source Screening for Learning Shared Feature Extractors

arXiv 2602.16125 ↗First posted 18 Feb 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Learning with shared representation is widely recognized as an effective way to separate commonalities from heterogeneity across various heterogeneous sources. Most existing work includes all related data sources via simultaneously training a common feature extractor and source-specific heads. It is well understood that data sources with low relevance or poor quality may hinder representation learning. In this paper, we further dive into the question of which data sources should be learned jointly by focusing on the traditionally deemed ``good'' collection of sources, in which individual sources have similar relevance and qualities with respect to the true underlying common structure. Towards tractability, we focus on the linear setting where sources share a low-dimensional subspace. We find that source screening can play a central role in statistically optimal subspace estimation. We show that, for a broad class of problem instances, training on a carefully selected subset of sources suffices to achieve minimax optimality, even when a substantial portion of data is discarded. We formalize the notion of an informative subpopulation, develop algorithms and practical heuristics for identifying such subsets, and validate their effectiveness through both theoretical analysis and empirical evaluations on synthetic and real-world datasets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 18 Feb 2026.

Builds on

33 of 55 references.

References · 33

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  6. 2019Advances and Open Problems in Federated Learningcited by 3,021
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  8. 2016Progressive Neural Networkscited by 1,332
  9. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  10. 2018LEAF: A Benchmark for Federated Settingscited by 780
  11. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  12. 2021Retiring Adult: New Datasets for Fair Machine Learningcited by 443
21 more
  1. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 374
  2. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  3. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 218
  4. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  5. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 196
  6. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  7. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  8. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  9. 2020Provable Meta-Learning of Linear Representationscited by 161
  10. 2020Optimal Client Sampling for Federated Learningcited by 123
  11. 2020A Survey on Negative Transfercited by 120
  12. 2024Parameter Competition Balancing for Model Mergingcited by 71
  13. 2021Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Samplingcited by 66
  14. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 55
  15. 2021How Fine-Tuning Allows for Effective Meta-Learningcited by 41
  16. 2022Adaptive and Robust Multi-Task Learningcited by 33
  17. 2023Learning from Similar Linear Representations: Adaptivity, Minimaxity, and Robustnesscited by 27
  18. 2020Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transferscited by 26
  19. 2022Subspace Recovery from Heterogeneous Data with Non-isotropic Noisecited by 14
  20. 2023Sample-Efficient Linear Representation Learning from Non-IID Non-Isotropic Datacited by 12
  21. 2023On the Benefits of Public Representations for Private Transfer Learning under Distribution Shiftcited by 5

Built on by

Not cited yet.