# FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extraction

arXiv: https://arxiv.org/abs/2407.19389 · HTML: https://mlsanity.com/paper/2407.19389/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 83 references that are in the map.
- First posted on arXiv 28 Jul 2024.

## Builds on (most cited first, 30 of 58)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients](https://mlsanity.com/paper/2010.01264/) (2020)
- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [Model Pruning Enables Efficient Federated Learning on Edge Devices](https://mlsanity.com/paper/1909.12326/) (2019)
- [Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning](https://mlsanity.com/paper/2101.11203/) (2021)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021)
- [Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization](https://mlsanity.com/paper/1902.05967/) (2019)
- [Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning](https://mlsanity.com/paper/2203.09249/) (2022)
- [Expanding the Reach of Federated Learning by Reducing Client Resource Requirements](https://mlsanity.com/paper/1812.07210/) (2018)
- [Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data](https://mlsanity.com/paper/2008.06180/) (2020)
- [Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradients](https://mlsanity.com/paper/2102.07053/) (2021)
- [Parameterized Knowledge Transfer for Personalized Federated Learning](https://mlsanity.com/paper/2111.02862/) (2021)
- [FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction](https://mlsanity.com/paper/2212.01548/) (2022)
- [On the Convergence of SGD with Biased Gradients](https://mlsanity.com/paper/2008.00051/) (2020)

## Built on by (most cited first, 21 of 21)

- [MM-OpenFGL: A Comprehensive Benchmark for Multimodal Federated Graph Learning](https://mlsanity.com/paper/2601.22416/) (2026)
- [FedEL: Federated Elastic Learning for Heterogeneous Devices](https://mlsanity.com/paper/2509.16902/) (2025)
- [HeteroTune: Efficient Federated Learning for Large Heterogeneous Models](https://mlsanity.com/paper/2411.16796/) (2024)
- [FedFetch: Faster Federated Learning with Adaptive Downstream Prefetching](https://mlsanity.com/paper/2504.15366/) (2025)
- [FedADP: Unified Model Aggregation for Federated Learning with Heterogeneous Model Architectures](https://mlsanity.com/paper/2505.06497/) (2025)
- [C^2Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learning](https://mlsanity.com/paper/2509.19674/) (2025)
- [On the Convergence and Stability of Distributed Sub-model Training](https://mlsanity.com/paper/2511.06132/) (2025)
- [CycleSL: Server-Client Cyclical Update Driven Scalable Split Learning](https://mlsanity.com/paper/2511.18611/) (2025)
- [FedRE: A Representation Entanglement Framework for Model-Heterogeneous Federated Learning](https://mlsanity.com/paper/2511.22265/) (2025)
- [Energy and Memory-Efficient Federated Learning With Ordered Layer Freezing](https://mlsanity.com/paper/2512.23200/) (2025)
- [Towards Robust Federated Multimodal Graph Learning under Modality Heterogeneity](https://mlsanity.com/paper/2605.12584/) (2026)
- [Towards Privacy-Preserving and Heterogeneity-aware Split Federated Learning via Probabilistic Masking](https://mlsanity.com/paper/2509.14603/) (2025)
- [UnifiedFL: A Dynamic Unified Learning Framework for Equitable Federation](https://mlsanity.com/paper/2510.26350/) (2025)
- [Representation-Aligned Multi-Scale Personalization for Federated Learning](https://mlsanity.com/paper/2604.11278/) (2026)
- [A Taxonomy and Resolution Strategy for Client-Level Disagreements in Federated Learning](https://mlsanity.com/paper/2604.23386/) (2026)
- [From Coordinate Matching to Structural Alignment: Rethinking Prototype Alignment in Heterogeneous Federated Learning](https://mlsanity.com/paper/2605.05959/) (2026)
- [HARMONY: Bridging the Personalization-Generalization Gap by Mitigating Representation Skew in Heterogeneous Split Federated Learning](https://mlsanity.com/paper/2605.07211/) (2026)
- [Capacity Confounds and Coverage Guarantees in Adaptive Sub-model Federated Learning](https://mlsanity.com/paper/2608.07157/) (2026)
- [FEAST: Federated Shared-Space Training for Resource-Heterogeneous Clients](https://mlsanity.com/paper/2608.09250/) (2026)
- [FedFIbOS: Fisher Importance based Optimal Submodelling for Heterogeneous Federated Learning](https://mlsanity.com/paper/2609.19559/) (2026)
- [FedSAP: Federated Learning with Structured Adaptive Partitioning for Multi-Domain Heterogeneous Edge Devices](https://mlsanity.com/paper/2610.01638/) (2026)

## Abstract (arXiv)

In federated learning (FL), accommodating clients' varied computational capacities poses a challenge, often limiting the participation of those with constrained resources in global model training. To address this issue, the concept of model heterogeneity through submodel extraction has emerged, offering a tailored solution that aligns the model's complexity with each client's computational capacity. In this work, we propose Federated Importance-Aware Submodel Extraction (FIARSE), a novel approach that dynamically adjusts submodels based on the importance of model parameters, thereby overcoming the limitations of previous static and dynamic submodel extraction methods. Compared to existing works, the proposed method offers a theoretical foundation for the submodel extraction and eliminates the need for additional information beyond the model parameters themselves to determine parameter importance, significantly reducing the overhead on clients. Extensive experiments are conducted on various datasets to showcase the superior performance of the proposed FIARSE.