mlsanityData to 7 Oct 2026

QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning

arXiv 2107.13892 ↗First posted 29 Jul 2021cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

Traditionally, federated learning (FL) aims to train a single global model while collaboratively using multiple clients and a server. Two natural challenges that FL algorithms face are heterogeneity in data across clients and collaboration of clients with em diverse resources. In this work, we introduce a textitquantized and textitpersonalized FL algorithm QuPeD that facilitates collective (personalized model compression) training via textitknowledge distillation (KD) among clients who have access to heterogeneous data and resources. For personalization, we allow clients to learn textitcompressed personalized models with different quantization parameters and model dimensions/structures. Towards this, first we propose an algorithm for learning quantized models through a relaxed optimization problem, where quantization values are also optimized over. When each client participating in the (federated) learning process has different requirements for the compressed model (both in model dimension and precision), we formulate a compressed personalization framework by introducing knowledge distillation loss for local client objectives collaborating through a global model. We develop an alternating proximal gradient update for solving this compressed personalization problem, and analyze its convergence properties. Numerically, we validate that QuPeD outperforms competing personalized FL methods, FedAvg, and local training of clients in various heterogeneous settings.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 33 of its 41 references that are in the map. First posted on arXiv 29 Jul 2021.

Builds on

33 of 41 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2018LEAF: A Benchmark for Federated Settingscited by 780
  7. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  8. 2017Federated Multi-Task Learningcited by 669
  9. 2017Deep Mutual Learningcited by 621
  10. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  11. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  12. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
21 more
  1. 2020Adaptive Personalized Federated Learningcited by 353
  2. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
  3. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  4. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  5. 2018Model compression via distillation and quantizationcited by 287
  6. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 234
  7. 2020Federated Learning of a Mixture of Global and Local Modelscited by 208
  8. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  9. 2020Binary Neural Networks: A Surveycited by 184
  10. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 179
  11. 2019Quantization Networkscited by 151
  12. 2020Lower Bounds and Optimal Algorithms for Personalized Federated Learningcited by 122
  13. 2017Training Quantized Nets: A Deeper Understandingcited by 74
  14. 2020Federated Mutual Learningcited by 70
  15. 2017Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMMcited by 60
  16. 2016Loss-aware Binarization of Deep Networkscited by 56
  17. 2018ProxQuant: Quantized Neural Networks via Proximal Operatorscited by 56
  18. 2020SQuARM-SGD: Communication-Efficient Momentum SGD for Decentralized Optimizationcited by 31
  19. 2018BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weightscited by 24
  20. 2020DBQ: A Differentiable Branch Quantizer for Lightweight Deep Neural Networkscited by 8
  21. 2021QuPeL: Quantized Personalization with Applications to Federated Learningcited by 3

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 8; 2023: 11; 2024: 8; 2025: 3.

32 papers.

Sort:

By area:

  1. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  2. 2022pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learningcited by 44
  3. 2022A Survey on Heterogeneous Federated Learningcited by 42
  4. 2023Recent Advances on Federated Learning: A Systematic Surveycited by 40
  5. 2023Efficient Personalized Federated Learning via Sparse Model-Adaptationcited by 38
  6. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 31
  7. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
  8. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  9. 2024Federated LoRA with Sparse Communicationcited by 26
  10. 2022FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregationcited by 21
  11. 2023PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guaranteescited by 15
  12. 2023Provable Data Subset Selection For Efficient Neural Network Trainingcited by 14
20 more
  1. 2024Federated Distillation: A Surveycited by 14
  2. 2023Complement Sparsification: Low-Overhead Model Pruning for Federated Learningcited by 12
  3. 2021SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learningcited by 11
  4. 2023Mixed-Precision Quantization for Federated Learning on Resource-Constrained Heterogeneous Devicescited by 11
  5. 2023FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Thingscited by 8
  6. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 8
  7. 2022An Empirical Study of Personalized Federated Learningcited by 6
  8. 2023FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learningcited by 5
  9. 2023Zone-based Federated Learning for Mobile Sensing Datacited by 4
  10. 2022Aligning Logits Generatively for Principled Black-Box Knowledge Distillationcited by 3
  11. 2024Unity is Power: Semi-Asynchronous Collaborative Training of Large-Scale Models with Structured Pruning in Resource-Limited Clientscited by 3
  12. 2022A Generative Framework for Personalized Learning and Estimation: Theory, Algorithms, and Privacycited by 2
  13. 2025Efficient Federated Learning with Timely Update Disseminationcited by 2
  14. 2024ADEPT: Hierarchical Bayes Approach to Personalized Federated Unsupervised Learningcited by 1
  15. 2024FedTrans: Efficient Federated Learning via Multi-Model Transformationcited by 1
  16. 2025Decentralized Optimization with Topology-Independent Communicationcited by 1
  17. 2023Heterogeneous Federated Learning Using Knowledge Codistillationcited by 0
  18. 2023Scalable Federated Learning for Clients with Different Input Image Sizes and Numbers of Output Categoriescited by 0
  19. 2024Architectural Blueprint For Heterogeneity-Resilient Federated Learningcited by 0
  20. 2025SPIRE: Conditional Personalization for Federated Diffusion Generative Modelscited by 0