mlsanityData to 7 Oct 2026

Federated Model Heterogeneous Matryoshka Representation Learning

arXiv 2406.00488 ↗First posted 1 Jun 2024cs.LG cs.DCcited by 21

Not on the map. Why?

Abstract (arXiv)

Model heterogeneous federated learning (MHeteroFL) enables FL clients to collaboratively train models with heterogeneous structures in a distributed fashion. However, existing MHeteroFL methods rely on training loss to transfer knowledge between the client model and the server model, resulting in limited knowledge exchange. To address this limitation, we propose the Federated model heterogeneous Matryoshka Representation Learning (FedMRL) approach for supervised learning tasks. It adds an auxiliary small homogeneous model shared by clients with heterogeneous local models. (1) The generalized and personalized representations extracted by the two models' feature extractors are fused by a personalized lightweight representation projector. This step enables representation fusion to adapt to local data distribution. (2) The fused representation is then used to construct Matryoshka representations with multi-dimensional and multi-granular embedded representations learned by the global homogeneous model header and the local heterogeneous model header. This step facilitates multi-perspective representation learning and improves model learning capability. Theoretical analysis shows that FedMRL achieves a O(1/T) non-convex convergence rate. Extensive experiments on benchmark datasets demonstrate its superior model accuracy with low communication and computational costs compared to seven state-of-the-art baselines. It achieves up to 8.48% and 24.94% accuracy improvement compared with the state-of-the-art and the best same-category baseline, respectively.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 47 of its 69 references that are in the map. First posted on arXiv 1 Jun 2024.

Builds on

47 of 69 references.

References · 47

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  3. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  4. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  5. 2021Towards Personalized Federated Learningcited by 462
  6. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  7. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  8. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  9. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  10. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
  11. 2022Matryoshka Representation Learningcited by 268
  12. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
35 more
  1. 2021Personalized Federated Learning using Hypernetworkscited by 210
  2. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  3. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  4. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 180
  5. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  6. 2022Federated Learning with Partial Model Personalizationcited by 114
  7. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 108
  8. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  9. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 91
  10. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  11. 2020Practical One-Shot Federated Learning for Cross-Silo Settingcited by 76
  12. 2020Federated Mutual Learningcited by 70
  13. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 67
  14. 2021Decentralized Federated Learning through Proxy Model Sharingcited by 56
  15. 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
  16. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  17. 2024FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learningcited by 47
  18. 2021Fed2: Feature-Aligned Federated Learningcited by 42
  19. 2019Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Datacited by 39
  20. 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
  21. 2023Towards Personalized Federated Learning via Heterogeneous Model Reassemblycited by 31
  22. 2022Architecture Agnostic Federated Learning for Neural Networkscited by 30
  23. 2020Byzantine-resilient Decentralized Stochastic Gradient Descentcited by 28
  24. 2021FedMatch: Federated Learning Over Heterogeneous Question Answering Datacited by 23
  25. 2022FedClassAvg: Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networkscited by 23
  26. 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learningcited by 21
  27. 2020Cooperative Learning via Federated Distillation over Fading Channelscited by 16
  28. 2022Lottery Aware Sparsity Hunting: Enabling Federated Learning on Resource-Limited Edgecited by 12
  29. 2022Enhancing Heterogeneous Federated Learning with Knowledge Extraction and Multi-Model Fusioncited by 10
  30. 2023FedSSA: Semantic Similarity-based Aggregation for Efficient Model-Heterogeneous Personalized Federated Learningcited by 10
  31. 2022Completely Heterogeneous Federated Learningcited by 9
  32. 2023pFedES: Model Heterogeneous Personalized Federated Learning with Feature Extractor Sharingcited by 9
  33. 2023Federated Learning via Input-Output Collaborative Distillationcited by 6
  34. 2023Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learningcited by 5
  35. 2024pFedAFM: Adaptive Feature Mixture for Batch-Level Personalization in Heterogeneous Federated Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 8.

21 papers.

Sort:

By area:

  1. 2025Can Textual Gradient Work in Federated Learning?cited by 14
  2. 2025HtFLlib: A Comprehensive Heterogeneous Federated Learning Library and Benchmarkcited by 7
  3. 2025Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilizationcited by 6
  4. 2025Federated Out-of-Distribution Generalization: A Causal Augmentation Viewcited by 4
  5. 2025Federated Deconfounding and Debiasing Learning for Out-of-Distribution Generalizationcited by 4
  6. 2025Enhancing Visual Representation with Textual Semantics: Textual Semantics-Powered Prototypes for Heterogeneous Federated Learningcited by 2
  7. 2025Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learningcited by 2
  8. 2025FedSKD: Aggregation-free Model-heterogeneous Federated Learning via Multi-dimensional Similarity Knowledge Distillation for Medical Image Classificationcited by 1
  9. 2025FedRE: A Representation Entanglement Framework for Model-Heterogeneous Federated Learningcited by 1
  10. 2026ForgeVLA: Federated Vision-Language-Action Learning without Language Annotationscited by 1
  11. 2026m3BERT: A Modern, Multi-lingual, Matryoshka Bidirectional Encodercited by 1
  12. 2024Adaptive Guidance for Local Training in Heterogeneous Federated Learningcited by 0
9 more
  1. 2025Towards Initialization-Agnostic Clustering with Iterative Adaptive Resonance Theorycited by 0
  2. 2025FedL2T: Personalized Federated Learning with Two-Teacher Distillation for Seizure Predictioncited by 0
  3. 2025Eigenfunction Extraction for Ordered Representation Learningcited by 0
  4. 2026Social Hippocampus Memory Learningcited by 0
  5. 2026FedDBP: Enhancing Federated Prototype Learning with Dual-Branch Features and Personalized Global Fusioncited by 0
  6. 2026From Selection to Scheduling: Federated Geometry-Aware Correction Makes Exemplar Replay Work Better under Continual Dynamic Heterogeneitycited by 0
  7. 2026From Coordinate Matching to Structural Alignment: Rethinking Prototype Alignment in Heterogeneous Federated Learningcited by 0
  8. 2026FedTopo: Relation-Level Topology Sharing for Model-Heterogeneous Federated Learningcited by 0
  9. 2026vFedProtoQNAS: Prototype-Guided Personalized Quantum Neural Architecture Search for Virtual Federated Learningcited by 0