mlsanityData to 7 Oct 2026

Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well

arXiv 2001.02312 ↗First posted 7 Jan 2020cs.LG stat.MLcited by 57

Not on the map. Why?

Abstract (arXiv)

We propose Stochastic Weight Averaging in Parallel (SWAP), an algorithm to accelerate DNN training. Our algorithm uses large mini-batches to compute an approximate solution quickly and then refines it by averaging the weights of multiple models computed independently and in parallel. The resulting models generalize equally well as those trained with small mini-batches but are produced in a substantially shorter time. We demonstrate the reduction in training time and the good generalization performance of the resulting models on the computer vision datasets CIFAR10, CIFAR100, and ImageNet.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 2; 2023: 11; 2024: 16; 2025: 16; 2026: 7.

57 papers.

Sort:

By area:

  1. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  2. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  3. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  4. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  5. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  6. 2024Knowledge Fusion of Large Language Modelscited by 110
  7. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
  8. 2023Deep Model Fusion: A Surveycited by 89
  9. 2021RobustART: Benchmarking Robustness on Architecture Design and Training Techniquescited by 78
  10. 2024Parameter Competition Balancing for Model Mergingcited by 71
  11. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 65
  12. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 53
38 more
  1. 2021Consensus Control for Decentralized Deep Learningcited by 49
  2. 2024LoRA Soups: Merging LoRAs for Practical Skill Composition Taskscited by 49
  3. 2023Randomized Adversarial Training via Taylor Expansioncited by 28
  4. 2023Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarizationcited by 28
  5. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 18
  6. 2024Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learningcited by 17
  7. 2024AlignGuard: Scalable Safety Alignment for Text-to-Image Generationcited by 17
  8. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  9. 2021Truly Sparse Neural Networks at Scalecited by 15
  10. 2021LocalNewton: Reducing Communication Bottleneck for Distributed Learningcited by 15
  11. 2024QT-DoG: Quantization-aware Training for Domain Generalizationcited by 13
  12. 2024Task Arithmetic Through The Lens Of One-Shot Federated Learningcited by 12
  13. 2023Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Datacited by 7
  14. 2025LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Mergingcited by 7
  15. 2025LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjointcited by 7
  16. 2024Knowledge Fusion of Chat LLMs: A Preliminary Technical Reportcited by 6
  17. 2024DynaMMo: Dynamic Model Merging for Efficient Class Incremental Learning for Medical Imagescited by 6
  18. 2025When, Where and Why to Average Weights?cited by 6
  19. 2025Model Unmerging: Making Your Models Unmergeable for Secure Model Sharingcited by 6
  20. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  21. 2025Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decouplingcited by 5
  22. 2025CLUES: Collaborative High-Quality Data Selection for LLMs via Training Dynamicscited by 4
  23. 2021Loss Landscape Dependent Self-Adjusting Learning Rates in Decentralized Stochastic Gradient Descentcited by 3
  24. 2023A Survey of Historical Learning: Learning Models with Learning Historycited by 3
  25. 2023Accelerating Large Batch Training via Gradient Signal to Noise Ratio (GSNR)cited by 3
  26. 2024ProFuser: Progressive Fusion of Large Language Modelscited by 3
  27. 2025Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Datacited by 3
  28. 2025Probabilistic Token Alignment for Large Language Model Fusioncited by 3
  29. 2026Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMscited by 3
  30. 2023Hierarchical Weight Averaging for Deep Neural Networkscited by 2
  31. 2023The Split Matters: Flat Minima Methods for Improving the Performance of GNNscited by 2
  32. 2025Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitaskingcited by 2
  33. 2025T3: Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysiscited by 2
  34. 2025Navigating the Accuracy-Size Trade-Off with Flexible Model Mergingcited by 1
  35. 2026Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimizationcited by 1
  36. 2024eXplainable Bayesian Multi-Perspective Generative Retrievalcited by 0
  37. 2025Multi-Level Collaboration in Model Mergingcited by 0
  38. 2025Efficient Multi-Task Modeling through Automated Fusion of Trained Modelscited by 0

50 most cited of 57 All 57 (JSON)