mlsanityData to 14 Oct 2026

Online Model Compression for Federated Learning with Large Models

arXiv 2205.03494 ↗First posted 6 May 2022cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

This paper addresses the challenges of training large neural network models under federated learning settings: high on-device memory usage and communication cost. The proposed Online Model Compression (OMC) provides a framework that stores model parameters in a compressed format and decompresses them only when needed. We use quantization as the compression method in this paper and propose three methods, (1) using per-variable transformation, (2) weight matrices only quantization, and (3) partial parameter quantization, to minimize the impact on model accuracy. According to our experiments on two recent neural networks for speech recognition and two different datasets, OMC can reduce memory usage and communication cost of model parameters by up to 59% while attaining comparable accuracy and training speed when compared with full-precision training.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 6 May 2022.

Builds on

22 of 27 references.

References · 22

  1. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  7. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  8. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  9. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  10. 2021A Field Guide to Federated Optimizationcited by 315
  11. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  12. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 163
10 more
  1. 2016Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruningcited by 134
  2. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  3. 2019Recognizing long-form speech using streaming end-to-end modelscited by 77
  4. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 69
  5. 2021Pareto-Optimal Quantized ResNet Is Mostly 4-bitcited by 26
  6. 20224-bit Conformer with Native Quantization Aware Training for Speech Recognitioncited by 24
  7. 2021Partial Variable Training for Efficient On-Device Federated Learningcited by 14
  8. 2021NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimizationcited by 13
  9. 2021Enabling On-Device Training of Speech Recognition Models with Federated Dropoutcited by 12
  10. 2019Distributed Fixed Point Methods with Compressed Iteratescited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 1; 2026: 1.

7 papers.

  1. 2023FedCode: Communication-Efficient Federated Learning via Transferring Codebookscited by 3
  2. 2024Vanishing Variance Problem in Fully Decentralized Neural-Network Systemscited by 3
  3. 2026Toward Efficient Membership Inference Attacks against Federated Large Language Models: A Projection Residual Approachcited by 3
  4. 2023Federated Automatic Differentiationcited by 2
  5. 2024A Resource-Adaptive Approach for Federated Learning under Resource-Constrained Environmentscited by 1
  6. 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insightscited by 1
  7. 2024Federated Learning of Large ASR Models in the Real Worldcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023FedSZ: Leveraging Error-Bounded Lossy Compression for Federated Learning Communications
  2. 2024Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codes
  3. 2021Communication-Efficient Federated Learning via Predictive Coding
  4. 2021DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning
  5. 2022Deep Hierarchy Quantization Compression algorithm based on Dynamic Sampling
  6. 2025FedSparQ: Adaptive Sparse Quantization with Error Feedback for Robust & Efficient Federated Learning
  7. 2022Lossy Gradient Compression: How Much Accuracy Can One Bit Buy?
  8. 2023Federated learning compression designed for lightweight communications