mlsanityData to 7 Oct 2026

Federated Learning with Intermediate Representation Regularization

arXiv 2210.15827 ↗First posted 28 Oct 2022cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

In contrast to centralized model training that involves data collection, federated learning (FL) enables remote clients to collaboratively train a model without exposing their private data. However, model performance usually degrades in FL due to the heterogeneous data generated by clients of diverse characteristics. One promising strategy to maintain good performance is by limiting the local training from drifting far away from the global model. Previous studies accomplish this by regularizing the distance between the representations learned by the local and global models. However, they only consider representations from the early layers of a model or the layer preceding the output layer. In this study, we introduce FedIntR, which provides a more fine-grained regularization by integrating the representations of intermediate layers into the local training process. Specifically, FedIntR computes a regularization term that encourages the closeness between the intermediate layer representations of the local and global models. Additionally, FedIntR automatically determines the contribution of each layer's representation to the regularization term based on the similarity between local and global representations. We conduct extensive experiments on various datasets to show that FedIntR can achieve equivalent or higher performance compared to the state-of-the-art approaches. Our code is available at https://github.com/YLTun/FedIntR.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 28 Oct 2022.

Builds on

32 of 41 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  8. 2019Advances and Open Problems in Federated Learningcited by 3,021
  9. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  10. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  11. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  12. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
20 more
  1. 2018Federated Learning with Non-IID Datacited by 1,099
  2. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  3. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  4. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  5. 2021Model-Contrastive Federated Learningcited by 762
  6. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  7. 2020Federated Learning with Matched Averagingcited by 564
  8. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  9. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
  10. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  11. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
  12. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  13. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  14. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  15. 2021Multi-Center Federated Learning: Clients Clustering for Better Personalizationcited by 112
  16. 2022Layer-wised Model Aggregation for Personalized Federated Learningcited by 95
  17. 2019Robust Federated Learning in a Heterogeneous Environmentcited by 88
  18. 2020FedGroup: Efficient Clustered Federated Learning via Decomposed Data-Driven Measurecited by 45
  19. 2021Intermediate Layers Matter in Momentum Contrastive Self Supervised Learningcited by 19
  20. 2021Compare Where It Matters: Using Layer-Wise Regularization To Improve Federated Learning on Heterogeneous Datacited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 1.

3 papers.

  1. 2023Contrastive encoder pre-training-based clustered federated learning for heterogeneous datacited by 5
  2. 2024Enhancing Object Detection with Hybrid dataset in Manufacturing Environments: Comparing Federated Learning to Conventional Techniquescited by 4
  3. 2025Regularized Federated Learning for Privacy-Preserving Dysarthric and Elderly Speech Recognitioncited by 1