# Architecture Agnostic Federated Learning for Neural Networks

arXiv: https://arxiv.org/abs/2202.07757 · HTML: https://mlsanity.com/paper/2202.07757/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 15 Feb 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Ditto: Fair and Robust Federated Learning Through Personalization](https://mlsanity.com/paper/2012.04221/) (2020)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [FedProto: Federated Prototype Learning across Heterogeneous Clients](https://mlsanity.com/paper/2105.00243/) (2021)
- [Improving Federated Learning Personalization via Model Agnostic Meta Learning](https://mlsanity.com/paper/1909.12488/) (2019)
- [No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2106.05001/) (2021)
- [Model Fusion via Optimal Transport](https://mlsanity.com/paper/1910.05653/) (2019)
- [Adaptive Gradient-Based Meta-Learning Methods](https://mlsanity.com/paper/1906.02717/) (2019)
- [FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning](https://mlsanity.com/paper/2009.01974/) (2020)
- [FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data](https://mlsanity.com/paper/2005.11418/) (2020)
- [Salvaging Federated Learning by Local Adaptation](https://mlsanity.com/paper/2002.04758/) (2020)
- [FedSplit: An algorithmic framework for fast federated optimization](https://mlsanity.com/paper/2005.05238/) (2020)
- [Towards Fair Federated Learning with Zero-Shot Data Augmentation](https://mlsanity.com/paper/2104.13417/) (2021)
- [Federated Learning via Synthetic Data](https://mlsanity.com/paper/2008.04489/) (2020)
- [Semi-Cyclic Stochastic Gradient Descent](https://mlsanity.com/paper/1904.10120/) (2019)
- [Improving Transformers with Probabilistic Attention Keys](https://mlsanity.com/paper/2110.08678/) (2021)
- [WAFFLE: Weighted Averaging for Personalized Federated Learning](https://mlsanity.com/paper/2110.06978/) (2021)

## Built on by (most cited first, 30 of 30)

- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [FedGH: Heterogeneous Federated Learning with Generalized Global Header](https://mlsanity.com/paper/2303.13137/) (2023)
- [Personalized Subgraph Federated Learning](https://mlsanity.com/paper/2206.10206/) (2022)
- [DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2309.13546/) (2023)
- [Federated Model Heterogeneous Matryoshka Representation Learning](https://mlsanity.com/paper/2406.00488/) (2024)
- [Federated Self-supervised Learning for Heterogeneous Clients](https://mlsanity.com/paper/2205.12493/) (2022)
- [FedSSA: Semantic Similarity-based Aggregation for Efficient Model-Heterogeneous Personalized Federated Learning](https://mlsanity.com/paper/2312.09006/) (2023)
- [Completely Heterogeneous Federated Learning](https://mlsanity.com/paper/2210.15865/) (2022)
- [pFedES: Model Heterogeneous Personalized Federated Learning with Feature Extractor Sharing](https://mlsanity.com/paper/2311.06879/) (2023)
- [Collaborative Learning via Prediction Consensus](https://mlsanity.com/paper/2305.18497/) (2023)
- [A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applications](https://mlsanity.com/paper/2403.01387/) (2024)
- [Decentralized Learning with Multi-Headed Distillation](https://mlsanity.com/paper/2211.15774/) (2022)
- [Personalised Federated Learning On Heterogeneous Feature Spaces](https://mlsanity.com/paper/2301.11447/) (2023)
- [Robust Asymmetric Heterogeneous Federated Learning with Corrupted Clients](https://mlsanity.com/paper/2503.09206/) (2025)
- [From Laboratory to Real World: A New Benchmark Towards Privacy-Preserved Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2503.12232/) (2025)
- [UNIDEAL: Curriculum Knowledge Distillation Federated Learning](https://mlsanity.com/paper/2309.08961/) (2023)
- [Privacy Preserving Bayesian Federated Learning in Heterogeneous Settings](https://mlsanity.com/paper/2306.07959/) (2023)
- [Secure and Fast Asynchronous Vertical Federated Learning via Cascaded Hybrid Optimization](https://mlsanity.com/paper/2306.16077/) (2023)
- [Federated Learning for Estimating Heterogeneous Treatment Effects](https://mlsanity.com/paper/2402.17705/) (2024)
- [Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments](https://mlsanity.com/paper/2505.19699/) (2025)
- [Toward Enhancing Representation Learning in Federated Multi-Task Settings](https://mlsanity.com/paper/2602.01626/) (2026)
- [Feature Correlation-guided Knowledge Transfer for Federated Self-supervised Learning](https://mlsanity.com/paper/2211.07364/) (2022)
- [A Kernel Perspective on Distillation-based Collaborative Learning](https://mlsanity.com/paper/2410.17592/) (2024)
- [FedRE: A Representation Entanglement Framework for Model-Heterogeneous Federated Learning](https://mlsanity.com/paper/2511.22265/) (2025)
- [Federated Learning of Neural ODE Models with Different Iteration Counts](https://mlsanity.com/paper/2208.09478/) (2022)
- [Global Layers: Non-IID Tabular Federated Learning](https://mlsanity.com/paper/2305.19290/) (2023)
- [Horizontal Federated Computer Vision](https://mlsanity.com/paper/2401.00390/) (2023)
- [Trust-free Personalized Decentralized Learning](https://mlsanity.com/paper/2410.11378/) (2024)
- [Federated Ensemble Learning with Progressive Model Personalization](https://mlsanity.com/paper/2602.14244/) (2026)
- [COSMOS: Model-Agnostic Personalized Federated Learning with Clustered Server Models and Pseudo-Label-Only Communication](https://mlsanity.com/paper/2605.11165/) (2026)

## Abstract (arXiv)

With growing concerns regarding data privacy and rapid increase in data volume, Federated Learning(FL) has become an important learning paradigm. However, jointly learning a deep neural network model in a FL setting proves to be a non-trivial task because of the complexities associated with the neural networks, such as varied architectures across clients, permutation invariance of the neurons, and presence of non-linear transformations in each layer. This work introduces a novel Federated Heterogeneous Neural Networks (FedHeNN) framework that allows each client to build a personalised model without enforcing a common architecture across clients. This allows each client to optimize with respect to local data and compute constraints, while still benefiting from the learnings of other (potentially more powerful) clients. The key idea of FedHeNN is to use the instance-level representations obtained from peer clients to guide the simultaneous training on each client. The extensive experimental results demonstrate that the FedHeNN framework is capable of learning better performing models on clients in both the settings of homogeneous and heterogeneous architectures across clients.