mlsanityData to 7 Oct 2026

Federated Representation Learning in the Under-Parameterized Regime

arXiv 2406.04596 ↗First posted 7 Jun 2024cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Federated representation learning (FRL) is a popular personalized federated learning (FL) framework where clients work together to train a common representation while retaining their personalized heads. Existing studies, however, largely focus on the over-parameterized regime. In this paper, we make the initial efforts to investigate FRL in the under-parameterized regime, where the FL model is insufficient to express the variations in all ground-truth models. We propose a novel FRL algorithm FLUTE, and theoretically characterize its sample complexity and convergence rate for linear models in the under-parameterized regime. To the best of our knowledge, this is the first FRL algorithm with provable performance guarantees in this regime. FLUTE features a data-independent random initialization and a carefully designed objective function that aids the distillation of subspace spanned by the global optimal representation from the misaligned local representations. On the technical side, we bridge low-rank matrix approximation techniques with the FL analysis, which may be of broad interest. We also extend FLUTE beyond linear representations. Experimental results demonstrate that FLUTE outperforms state-of-the-art FRL solutions in both synthetic and real-world tasks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 37 of its 51 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

37 of 51 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2019Searching for MobileNetV3cited by 1,984
  7. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  8. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  9. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  10. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  11. 2019Federated Learning with Personalization Layerscited by 533
  12. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 509
25 more
  1. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 408
  2. 2017Deep Models Under the GAN: Information Leakage from Collaborative Deep Learningcited by 326
  3. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
  4. 2018Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learningcited by 242
  5. 2021Mobile-Former: Bridging MobileNet and Transformercited by 234
  6. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  7. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  8. 2020Provable Meta-Learning of Linear Representationscited by 161
  9. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  10. 2017No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysiscited by 127
  11. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  12. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  13. 2021Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorizationcited by 44
  14. 2023FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policycited by 36
  15. 2022Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorizationcited by 34
  16. 2021Sample Efficient Linear Meta-Learning by Alternating Minimizationcited by 25
  17. 2017The Global Optimization Geometry of Low-Rank Matrix Optimizationcited by 24
  18. 2016A Unified Computational and Statistical Framework for Nonconvex Low-Rank Matrix Estimationcited by 21
  19. 2022Parameters or Privacy: A Provable Tradeoff Between Overparameterization and Membership Inferencecited by 14
  20. 2022Subspace Recovery from Heterogeneous Data with Non-isotropic Noisecited by 14
  21. 2023Share Your Representation Only: Guaranteed Improvement of the Privacy-Utility Tradeoff in Federated Learningcited by 13
  22. 2023Sample-Efficient Linear Representation Learning from Non-IID Non-Isotropic Datacited by 12
  23. 2022FedDAR: Federated Domain-Aware Representation Learningcited by 8
  24. 2016Distributed Multi-Task Learning with Shared Representationcited by 5
  25. 2022Efficient Convolutional Neural Networks on Raspberry Pi for Image Classificationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2025Federated Learning for Medical Image Classification: A Comprehensive Benchmarkcited by 6
  2. 2024F3OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristicscited by 4
  3. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
  4. 2024FedLog: Personalized Federated Classification with Less Communication and More Flexibilitycited by 0
  5. 2025ATR-Bench: A Federated Learning Benchmark for Adaptation, Trust, and Reasoningcited by 0
  6. 2026Heterogeneity-Aware Client Selection Methodology For Efficient Federated Learningcited by 0
  7. 2026FedOBP: Federated Optimal Brain Personalization through Cloud-Edge Element-wise Decouplingcited by 0