mlsanityData to 14 Oct 2026

Federated Deep Equilibrium Learning: Harnessing Compact Global Representations to Enhance Personalization

arXiv 2309.15659 ↗First posted 27 Sep 2023cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) has emerged as a groundbreaking distributed learning paradigm enabling clients to train a global model collaboratively without exchanging data. Despite enhancing privacy and efficiency in information retrieval and knowledge management contexts, training and deploying FL models confront significant challenges such as communication bottlenecks, data heterogeneity, and memory limitations. To comprehensively address these challenges, we introduce FeDEQ, a novel FL framework that incorporates deep equilibrium learning and consensus optimization to harness compact global data representations for efficient personalization. Specifically, we design a unique model structure featuring an equilibrium layer for global representation extraction, followed by explicit layers tailored for local personalization. We then propose a novel FL algorithm rooted in the alternating directions method of multipliers (ADMM), which enables the joint optimization of a shared equilibrium layer and individual personalized layers across distributed datasets. Our theoretical analysis confirms that FeDEQ converges to a stationary point, achieving both compact global representations and optimal personalized parameters for each client. Extensive experiments on various benchmarks demonstrate that FeDEQ matches the performance of state-of-the-art personalized FL methods, while significantly reducing communication size by up to 4 times and memory footprint by 1.5 times during training.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 55 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

34 of 55 references.

References · 34

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2019Advances and Open Problems in Federated Learningcited by 3,046
  3. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  4. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  5. 2018Federated Learning with Non-IID Datacited by 1,108
  6. 2018LEAF: A Benchmark for Federated Settingscited by 792
  7. 2017Federated Multi-Task Learningcited by 672
  8. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  9. 2018Universal Transformerscited by 647
  10. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  11. 2019Deep Equilibrium Modelscited by 615
  12. 2019Federated Learning with Personalization Layerscited by 541
22 more
  1. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  2. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  3. 2020Adaptive Personalized Federated Learningcited by 358
  4. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  5. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 269
  6. 2020Federated Learning of a Mixture of Global and Local Modelscited by 210
  7. 2021Efficient and Modular Implicit Differentiationcited by 203
  8. 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 192
  9. 2016Distributed Mean Estimation with Limited Communicationcited by 136
  10. 2019Implicit Deep Learningcited by 124
  11. 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
  12. 2021JFB: Jacobian-Free Backpropagation for Implicit Networkscited by 86
  13. 2021Federated Reconstruction: Partially Local Federated Learningcited by 79
  14. 2021Personalized Federated Learning with Gaussian Processescited by 76
  15. 2014Convergence Analysis of Alternating Direction Method of Multipliers for a Family of Nonconvex Problemscited by 64
  16. 2021Personalized Federated Learning through Local Memorizationcited by 57
  17. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 56
  18. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  19. 2021Personalized Federated Learning: A Unified Framework and Universal Optimization Techniquescited by 34
  20. 2021FedJAX: Federated learning simulation with JAXcited by 32
  21. 2022Motley: Benchmarking Heterogeneity and Personalization in Federated Learningcited by 31
  22. 2020Inexact-ADMM Based Federated Meta-Learning for Fast and Continual Edge Learningcited by 22

Built on by

Not cited yet.