mlsanityData to 14 Oct 2026

Layer-wise Quantization for Quantized Optimistic Dual Averaging

arXiv 2505.14371 ↗First posted 20 May 2025cs.LG math.OCcited by 1

Not on the map. Why?

Abstract (arXiv)

Modern deep neural networks exhibit heterogeneity across numerous layers of various types such as residuals, multi-head attention, etc., due to varying structures (dimensions, activation functions, etc.), distinct representation characteristics, which impact predictions. We develop a general layer-wise quantization framework with tight variance and code-length bounds, adapting to the heterogeneities over the course of training. We then apply a new layer-wise quantization technique within distributed variational inequalities (VIs), proposing a novel Quantized Optimistic Dual Averaging (QODA) algorithm with adaptive learning rates, which achieves competitive convergence rates for monotone VIs. We empirically show that QODA achieves up to a 150% speedup over the baselines in end-to-end training time for training Wasserstein GAN on 12+ GPUs.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 48 of its 97 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

48 of 97 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2019Advances and Open Problems in Federated Learningcited by 3,046
  6. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  7. 2018Adversarially Robust Generalization Requires More Datacited by 318
  8. 2017Training GANs with Optimismcited by 217
  9. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  10. 2015Fast Convergence of Regularized Learning in Gamescited by 150
  11. 2018A Variational Inequality Perspective on Generative Adversarial Networkscited by 144
  12. 2019A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approachcited by 143
36 more
  1. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 136
  2. 2018Optimistic mirror descent in saddle-point problems: Going the extra (gradient) milecited by 136
  3. 2019Distributed Learning with Compressed Gradient Differencescited by 112
  4. 2019Reducing Noise in GAN Training with Variance Reduced Extragradientcited by 62
  5. 2020Efficiently Solving MDPs with Stochastic Mirror Descentcited by 53
  6. 2018Online Adaptive Methods, Universality and Accelerationcited by 51
  7. 2021Adaptive Learning in Continuous Games: Optimal Regret Bounds and Convergence to Nash Equilibriumcited by 51
  8. 2020Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize Scalingcited by 49
  9. 2020Robust Reinforcement Learning via Adversarial training with Langevin Dynamicscited by 45
  10. 2020Adaptive Gradient Quantization for Data-Parallel SGDcited by 45
  11. 2018Golden Ratio Algorithms for Variational Inequalitiescited by 38
  12. 2022Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methodscited by 37
  13. 2017Extragradient method with variance reduction for stochastic variational inequalitiescited by 33
  14. 2019On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learningcited by 32
  15. 2021The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communicationcited by 31
  16. 2015Projected Reflected Gradient Methods for Monotone Variational Inequalitiescited by 27
  17. 2019A Universal Algorithm for Variational Inequalities Adaptive to Smoothness and Noisecited by 26
  18. 2020Finite-Time Last-Iterate Convergence for Multi-Agent Learning in Gamescited by 26
  19. 2019NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantizationcited by 18
  20. 2022No-Regret Learning in Games with Noisy Feedback: Faster Rates and Adaptivity via Learning Rate Separationcited by 17
  21. 2022Adaptive Compression for Communication-Efficient Distributed Trainingcited by 16
  22. 2021Distributed Methods with Compressed Communication for Solving Variational Inequalities, with Theoretical Guaranteescited by 15
  23. 2022Optimal Algorithms for Decentralized Stochastic Variational Inequalitiescited by 14
  24. 2019Training GANs with Centripetal Accelerationcited by 13
  25. 2020Adaptive and Universal Algorithms for Variational Inequalities with Optimal Convergencecited by 12
  26. 2021IntSGD: Adaptive Floatless Compression of Stochastic Gradientscited by 9
  27. 2021Theoretically Better and Numerically Faster Distributed Optimization with Smoothness-Aware Quantization Techniquescited by 9
  28. 2024EfQAT: An Efficient Framework for Quantization-Aware Trainingcited by 7
  29. 2020Towards Tight Communication Lower Bounds for Distributed Optimisationcited by 6
  30. 2023Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimizationcited by 6
  31. 2022MixTailor: Mixed Gradient Aggregation for Robust Learning Against Tailored Attackscited by 5
  32. 2021CGX: Adaptive System Support for Communication-Efficient Deep Learningcited by 4
  33. 2023Distributed Extra-gradient with Optimal Complexity and Communication Guaranteescited by 2
  34. 2023Byzantine-Tolerant Methods for Distributed Variational Inequalitiescited by 2
  35. 2023Kimad: Adaptive Gradient Compression with Bandwidth Awarenesscited by 2
  36. 2023Bregman Proximal Method for Efficient Communications under Similaritycited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Modelscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2022Vertical Layering of Quantized Neural Networks for Heterogeneous Inference
  2. 2026SigmaQuant: Hardware-Aware Heterogeneous Quantization Method for Edge DNN Inference
  3. 2022QFT: Post-training quantization via fast joint finetuning of all degrees of freedom
  4. 2022Edge Inference with Fully Differentiable Quantized Mixed Precision Neural Networks
  5. 2025Adaptive Distribution-aware Quantization for Mixed-Precision Neural Networks
  6. 2021A Layer-wise Adversarial-aware Quantization Optimization for Improving Robustness
  7. 2024QGen: On the Ability to Generalize in Quantization Aware Training
  8. 2022Neural Networks with Quantization Constraints