# Federated Model Heterogeneous Matryoshka Representation Learning

arXiv: https://arxiv.org/abs/2406.00488 · HTML: https://mlsanity.com/paper/2406.00488/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 69 references that are in the map.
- First posted on arXiv 1 Jun 2024.

## Builds on (most cited first, 30 of 47)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [FedProto: Federated Prototype Learning across Heterogeneous Clients](https://mlsanity.com/paper/2105.00243/) (2021)
- [HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients](https://mlsanity.com/paper/2010.01264/) (2020)
- [Think Locally, Act Globally: Federated Learning with Local and Global Representations](https://mlsanity.com/paper/2001.01523/) (2020)
- [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022)
- [Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data](https://mlsanity.com/paper/1811.11479/) (2018)
- [Personalized Federated Learning using Hypernetworks](https://mlsanity.com/paper/2103.04628/) (2021)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [FedKD: Communication Efficient Federated Learning via Knowledge Distillation](https://mlsanity.com/paper/2108.13323/) (2021)
- [FedBABU: Towards Enhanced Representation for Federated Image Classification](https://mlsanity.com/paper/2106.06042/) (2021)
- [Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data](https://mlsanity.com/paper/2008.06180/) (2020)
- [Federated Learning with Partial Model Personalization](https://mlsanity.com/paper/2204.03809/) (2022)
- [Parameterized Knowledge Transfer for Personalized Federated Learning](https://mlsanity.com/paper/2111.02862/) (2021)
- [FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction](https://mlsanity.com/paper/2212.01548/) (2022)
- [Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer](https://mlsanity.com/paper/1912.11279/) (2019)
- [Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning](https://mlsanity.com/paper/2204.12703/) (2022)
- [Practical One-Shot Federated Learning for Cross-Silo Setting](https://mlsanity.com/paper/2010.01017/) (2020)
- [Federated Mutual Learning](https://mlsanity.com/paper/2006.16765/) (2020)
- [FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning](https://mlsanity.com/paper/2102.02514/) (2021)
- [Decentralized Federated Learning through Proxy Model Sharing](https://mlsanity.com/paper/2111.11343/) (2021)
- [FedGH: Heterogeneous Federated Learning with Generalized Global Header](https://mlsanity.com/paper/2303.13137/) (2023)
- [FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion](https://mlsanity.com/paper/2110.11027/) (2021)
- [FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning](https://mlsanity.com/paper/2401.03230/) (2024)
- [Fed2: Feature-Aligned Federated Learning](https://mlsanity.com/paper/2111.14248/) (2021)

## Built on by (most cited first, 21 of 21)

- [Can Textual Gradient Work in Federated Learning?](https://mlsanity.com/paper/2502.19980/) (2025)
- [HtFLlib: A Comprehensive Heterogeneous Federated Learning Library and Benchmark](https://mlsanity.com/paper/2506.03954/) (2025)
- [Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization](https://mlsanity.com/paper/2509.26520/) (2025)
- [Federated Out-of-Distribution Generalization: A Causal Augmentation View](https://mlsanity.com/paper/2504.19882/) (2025)
- [Federated Deconfounding and Debiasing Learning for Out-of-Distribution Generalization](https://mlsanity.com/paper/2505.04979/) (2025)
- [Enhancing Visual Representation with Textual Semantics: Textual Semantics-Powered Prototypes for Heterogeneous Federated Learning](https://mlsanity.com/paper/2503.13543/) (2025)
- [Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learning](https://mlsanity.com/paper/2509.22700/) (2025)
- [FedSKD: Aggregation-free Model-heterogeneous Federated Learning via Multi-dimensional Similarity Knowledge Distillation for Medical Image Classification](https://mlsanity.com/paper/2503.18981/) (2025)
- [FedRE: A Representation Entanglement Framework for Model-Heterogeneous Federated Learning](https://mlsanity.com/paper/2511.22265/) (2025)
- [ForgeVLA: Federated Vision-Language-Action Learning without Language Annotations](https://mlsanity.com/paper/2605.07474/) (2026)
- [m3BERT: A Modern, Multi-lingual, Matryoshka Bidirectional Encoder](https://mlsanity.com/paper/2605.19568/) (2026)
- [Adaptive Guidance for Local Training in Heterogeneous Federated Learning](https://mlsanity.com/paper/2410.06490/) (2024)
- [Towards Initialization-Agnostic Clustering with Iterative Adaptive Resonance Theory](https://mlsanity.com/paper/2505.04440/) (2025)
- [FedL2T: Personalized Federated Learning with Two-Teacher Distillation for Seizure Prediction](https://mlsanity.com/paper/2510.08984/) (2025)
- [Eigenfunction Extraction for Ordered Representation Learning](https://mlsanity.com/paper/2510.24672/) (2025)
- [Social Hippocampus Memory Learning](https://mlsanity.com/paper/2603.25614/) (2026)
- [FedDBP: Enhancing Federated Prototype Learning with Dual-Branch Features and Personalized Global Fusion](https://mlsanity.com/paper/2603.29455/) (2026)
- [From Selection to Scheduling: Federated Geometry-Aware Correction Makes Exemplar Replay Work Better under Continual Dynamic Heterogeneity](https://mlsanity.com/paper/2604.08617/) (2026)
- [From Coordinate Matching to Structural Alignment: Rethinking Prototype Alignment in Heterogeneous Federated Learning](https://mlsanity.com/paper/2605.05959/) (2026)
- [FedTopo: Relation-Level Topology Sharing for Model-Heterogeneous Federated Learning](https://mlsanity.com/paper/2607.26801/) (2026)
- [vFedProtoQNAS: Prototype-Guided Personalized Quantum Neural Architecture Search for Virtual Federated Learning](https://mlsanity.com/paper/2610.01718/) (2026)

## Abstract (arXiv)

Model heterogeneous federated learning (MHeteroFL) enables FL clients to collaboratively train models with heterogeneous structures in a distributed fashion. However, existing MHeteroFL methods rely on training loss to transfer knowledge between the client model and the server model, resulting in limited knowledge exchange. To address this limitation, we propose the Federated model heterogeneous Matryoshka Representation Learning (FedMRL) approach for supervised learning tasks. It adds an auxiliary small homogeneous model shared by clients with heterogeneous local models. (1) The generalized and personalized representations extracted by the two models' feature extractors are fused by a personalized lightweight representation projector. This step enables representation fusion to adapt to local data distribution. (2) The fused representation is then used to construct Matryoshka representations with multi-dimensional and multi-granular embedded representations learned by the global homogeneous model header and the local heterogeneous model header. This step facilitates multi-perspective representation learning and improves model learning capability. Theoretical analysis shows that FedMRL achieves a O(1/T) non-convex convergence rate. Extensive experiments on benchmark datasets demonstrate its superior model accuracy with low communication and computational costs compared to seven state-of-the-art baselines. It achieves up to 8.48% and 24.94% accuracy improvement compared with the state-of-the-art and the best same-category baseline, respectively.