Energizing Federated Learning via Filter-Aware Attention
Not on the map. Why?
Abstract (arXiv)
Federated learning (FL) is a promising distributed paradigm, eliminating the need for data sharing but facing challenges from data heterogeneity. Personalized parameter generation through a hypernetwork proves effective, yet existing methods fail to personalize local model structures. This leads to redundant parameters struggling to adapt to diverse data distributions. To address these limitations, we propose FedOFA, utilizing personalized orthogonal filter attention for parameter recalibration. The core is the Two-stream Filter-aware Attention (TFA) module, meticulously designed to extract personalized filter-aware attention maps, incorporating Intra-Filter Attention (IntraFa) and Inter-Filter Attention (InterFA) streams. These streams enhance representation capability and explore optimal implicit structures for local models. Orthogonal regularization minimizes redundancy by averting inter-correlation between filters. Furthermore, we introduce an Attention-Guided Pruning Strategy (AGPS) for communication efficiency. AGPS selectively retains crucial neurons while masking redundant ones, reducing communication costs without performance sacrifice. Importantly, FedOFA operates on the server side, incurring no additional computational cost on the client, making it advantageous in communication-constrained scenarios. Extensive experiments validate superior performance over state-of-the-art approaches, with code availability upon paper acceptance.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 18 Nov 2023.
Builds on
26 of 38 references.
References · 26
- 2017Attention Is All You Needcited by 60,106
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,653
- 2017Squeeze-and-Excitation Networkscited by 4,772
- 2018Federated Optimization in Heterogeneous Networkscited by 2,981
- 2018CBAM: Convolutional Block Attention Modulecited by 2,469
- 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 903
- 2021Model-Contrastive Federated Learningcited by 766
- 2020Personalized Federated Learning with Moreau Envelopescited by 616
- 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 570
- 2019Federated Learning with Personalization Layerscited by 539
14 more
- 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
- 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 324
- 2021FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Spacecited by 287
- 2021Coordinate Attention for Efficient Mobile Network Designcited by 240
- 2021Personalized Federated Learning using Hypernetworkscited by 211
- 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 187
- 2020Learning to Branch for Multi-Task Learningcited by 115
- 2022Layer-wised Model Aggregation for Personalized Federated Learningcited by 96
- 2021SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Trainingcited by 45
- 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
- 2022Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?cited by 16
- 2022Hypernetwork-based Personalized Federated Learning for Multi-Institutional CT Imagingcited by 14
- 2021MANAS: Multi-Scale and Multi-Level Neural Architecture Search for Low-Dose CT Denoisingcited by 3
- 2022Personalized Federated Hypernetworks for Privacy Preservation in Multi-Task Reinforcement Learningcited by 1
Built on by
Not cited yet.