mlsanityData to 7 Oct 2026

SWALP : Stochastic Weight Averaging in Low-Precision Training

arXiv 1904.11943 ↗First posted 26 Apr 2019cs.LG cs.AI stat.MLcited by 40

Not on the map. Why?

Abstract (arXiv)

Low precision operations can provide scalability, memory savings, portability, and energy efficiency. This paper proposes SWALP, an approach to low precision training that averages low-precision SGD iterates with a modified learning rate schedule. SWALP is easy to implement and can match the performance of full-precision SGD even with all numbers quantized down to 8 bits, including the gradient accumulators. Additionally, we show that SWALP converges arbitrarily close to the optimal solution for quadratic objectives, and to a noise ball asymptotically smaller than low precision SGD in strongly convex settings.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 7; 2022: 10; 2023: 11; 2024: 7; 2025: 5.

40 papers.

Sort:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 175
  3. 2023Deep Model Fusion: A Surveycited by 89
  4. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 65
  5. 2022Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentationcited by 41
  6. 2023Weighted Ensemble Models Are Strong Continual Learnerscited by 41
  7. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  8. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 36
  9. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  10. 2022On Distributed Adaptive Optimization with Gradient Compressioncited by 19
  11. 2021Assessments of epistemic uncertainty using Gaussian stochastic weight averaging for fluid-flow regressioncited by 15
  12. 2022DarKnight: An Accelerated Framework for Privacy and Integrity Preserving Deep Learning Using Trusted Hardwarecited by 13
28 more
  1. 2022Low-Precision Stochastic Gradient Langevin Dynamicscited by 12
  2. 2024Task Arithmetic Through The Lens Of One-Shot Federated Learningcited by 12
  3. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  4. 2022Low-Precision Arithmetic for Fast Gaussian Processescited by 10
  5. 2022Stability Based Generalization Bounds for Exponential Family Langevin Dynamicscited by 8
  6. 2023Dynamic Stashing Quantization for Efficient Transformer Trainingcited by 6
  7. 2025When, Where and Why to Average Weights?cited by 6
  8. 2021Whole Brain Segmentation with Full Volume Neural Networkcited by 5
  9. 2025Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptationcited by 5
  10. 2023Number Systems for Deep Neural Network Architectures: A Surveycited by 4
  11. 2023Tango: rethinking quantization for graph neural network training on GPUscited by 4
  12. 2021Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precisioncited by 3
  13. 2021Perturbated Gradients Updating within Unit Space for Deep Learningcited by 3
  14. 2023A Survey of Historical Learning: Learning Models with Learning Historycited by 3
  15. 2023Training with Mixed-Precision Floating-Point Assignmentscited by 2
  16. 2023Hierarchical Weight Averaging for Deep Neural Networkscited by 2
  17. 2023The Split Matters: Flat Minima Methods for Improving the Performance of GNNscited by 2
  18. 2021Noisy Truncated SGD: Optimization and Generalizationcited by 1
  19. 2022ST-CoNAL: Consistency-Based Acquisition Criterion Using Temporal Self-Ensemble for Active Learningcited by 1
  20. 2022Perturbed Iterate SGD for Lipschitz Continuous Loss Functions with Numerical Error and Adaptive Step Sizescited by 1
  21. 2024In-Distribution Consistency Regularization Improves the Generalization of Quantization-Aware Trainingcited by 1
  22. 2024On Exact Bit-level Reversible Transformers Without Changing Architecturescited by 1
  23. 2025Beyond Deepfake vs Real: Facial Deepfake Detection in the Open-Set Paradigmcited by 1
  24. 2021SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Trainingcited by 0
  25. 2021SAE: Sequential Anchored Ensemblescited by 0
  26. 2023Learn from the Past: A Proxy Guided Adversarial Defense Framework with Self Distillation Regularizationcited by 0
  27. 2024VIPriors 4: Visual Inductive Priors for Data-Efficient Deep Learning Challengescited by 0
  28. 2025Data-Efficient Challenges in Visual Inductive Priors: A Retrospectivecited by 0