mlsanityData to 7 Oct 2026

FedPerfix: Towards Partial Model Personalization of Vision Transformers in Federated Learning

arXiv 2308.09160 ↗First posted 17 Aug 2023cs.CV cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Personalized Federated Learning (PFL) represents a promising solution for decentralized learning in heterogeneous data environments. Partial model personalization has been proposed to improve the efficiency of PFL by selectively updating local model parameters instead of aggregating all of them. However, previous work on partial model personalization has mainly focused on Convolutional Neural Networks (CNNs), leaving a gap in understanding how it can be applied to other popular models such as Vision Transformers (ViTs). In this work, we investigate where and how to partially personalize a ViT model. Specifically, we empirically evaluate the sensitivity to data distribution of each type of layer. Based on the insights that the self-attention layer and the classification head are the most sensitive parts of a ViT, we propose a novel approach called FedPerfix, which leverages plugins to transfer information from the aggregated model to the local client as a personalization. Finally, we evaluate the proposed approach on CIFAR-100, OrganAMNIST, and Office-Home datasets and demonstrate its effectiveness in improving the model's performance compared to several advanced PFL methods.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 37 of its 47 references that are in the map. First posted on arXiv 17 Aug 2023.

Builds on

37 of 47 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  12. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
25 more
  1. 2021Video Swin Transformercited by 1,090
  2. 2016An overview of gradient descent optimization algorithmscited by 1,021
  3. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  4. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  5. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
  6. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  7. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  8. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  9. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  10. 2019Federated Learning with Personalization Layerscited by 533
  11. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  12. 2020Adaptive Personalized Federated Learningcited by 353
  13. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  14. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
  15. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  16. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 282
  17. 2021Personalized Federated Learning using Hypernetworkscited by 210
  18. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  19. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 180
  20. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  21. 2022Federated Learning with Partial Model Personalizationcited by 114
  22. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  23. 2022Towards a Unified View on Visual Parameter-Efficient Transfer Learningcited by 25
  24. 2022PGFed: Personalize Each Client's Global Objective for Federated Learningcited by 13
  25. 2022Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 9; 2026: 4.

21 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmarkcited by 21
  2. 2024Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Modelscited by 16
  3. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  4. 2024Towards Multi-modal Transformers in Federated Learningcited by 12
  5. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 10
  6. 2025Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Modelscited by 5
  7. 2024Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Spatiotemporal Modelingcited by 4
  8. 2025pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Modelscited by 3
  9. 2025pMixFed: Efficient Personalized Federated Learning through Adaptive Layer-Wise Mixupcited by 2
  10. 2025TAP: Two-Stage Adaptive Personalization of Multi-Task and Multi-Modal Foundation Models in Federated Learningcited by 2
  11. 2024Learn What You Need in Personalized Federated Learningcited by 1
  12. 2024Research on Personalized Compression Algorithm for Pre-trained Models Based on Homomorphic Entropy Increasecited by 1
9 more
  1. 2025FedAPM: Federated Learning via ADMM with Partial Model Personalizationcited by 1
  2. 2025FedRE: A Representation Entanglement Framework for Model-Heterogeneous Federated Learningcited by 1
  3. 2026OmniFM: Toward Modality-Robust and Task-Agnostic Federated Learning for Heterogeneous Medical Imagingcited by 1
  4. 2025Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selectioncited by 0
  5. 2025Generalizable Federated Learning using Client Adaptive Focal Modulationcited by 0
  6. 2025Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptationcited by 0
  7. 2026Federated Vision Transformer with Adaptive Focal Loss for Medical Image Classificationcited by 0
  8. 2026HiLoRA: Hierarchical Low-Rank Adaptation for Personalized Federated Learningcited by 0
  9. 2026MagicPrompt: Ultra-Lightweight Prompt Tuning for Video Generationcited by 0