mlsanityData to 7 Oct 2026

PopulAtion Parameter Averaging (PAPA)

arXiv 2304.03094 ↗First posted 6 Apr 2023cs.LG cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

Ensemble methods combine the predictions of multiple models to improve performance, but they require significantly higher computation costs at inference time. To avoid these costs, multiple neural networks can be combined into one by averaging their weights. However, this usually performs significantly worse than ensembling. Weight averaging is only beneficial when different enough to benefit from combining them, but similar enough to average well. Based on this idea, we propose PopulAtion Parameter Averaging (PAPA): a method that combines the generality of ensembling with the efficiency of weight averaging. PAPA leverages a population of diverse models (trained on different data orders, augmentations, and regularizations) while slowly pushing the weights of the networks toward the population average of the weights. We also propose PAPA variants (PAPA-all, and PAPA-2) that average weights rarely rather than continuously; all methods increase generalization, but PAPA tends to perform best. PAPA reduces the performance gap between averaging and ensembling, increasing the average accuracy of a population of models by up to 0.8% on CIFAR-10, 1.9% on CIFAR-100, and 1.6% on ImageNet when compared to training independent (non-averaged) models.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 49 of its 70 references that are in the map. First posted on arXiv 6 Apr 2023.

Builds on

49 of 70 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
37 more
  1. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  3. 2017Random Erasing Data Augmentationcited by 1,235
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  5. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  6. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  7. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  8. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  9. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  10. 2020Federated Learning with Matched Averagingcited by 564
  11. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  12. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  13. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  14. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  15. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  16. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  17. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  18. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  19. 2019Model Fusion via Optimal Transportcited by 217
  20. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  21. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  22. 2014EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimizationcited by 160
  23. 2017Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learningcited by 148
  24. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  25. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  26. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  27. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 85
  28. 2018The Unusual Effectiveness of Averaging in GAN Trainingcited by 73
  29. 2017Beyond RGB: Very High Resolution Urban Remote Sensing With Multimodal Deep Networkscited by 64
  30. 2022Re-basin via implicit Sinkhorn differentiationcited by 61
  31. 2017Between-class Learning for Image Classificationcited by 50
  32. 2019Deep Reinforcement Learning using Genetic Algorithm for Parameter Optimizationcited by 23
  33. 2023DART: Diversify-Aggregate-Repeat Training Improves Generalization of Neural Networkscited by 20
  34. 2019Wasserstein Adversarial Regularization (WAR) on label noisecited by 16
  35. 2015Experiments on Parallel Training of Deep Neural Network using Model Averagingcited by 14
  36. 2018Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networkscited by 13
  37. 2018Logistic Ensemble Modelscited by 1

Built on by

Cited by year

Citing papers per year: 2023: 6; 2024: 9; 2025: 8.

23 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  3. 2023Deep Model Fusion: A Surveycited by 89
  4. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  5. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 36
  6. 2025Model Merging in Pre-training of Large Language Modelscited by 36
  7. 2023Transformer Fusion with Optimal Transportcited by 35
  8. 2024Checkpoint Merging via Bayesian Optimization in LLM Pretrainingcited by 27
  9. 2023Sparse Model Soups: A Recipe for Improved Pruning via Model Averagingcited by 20
  10. 2024A Second-Order Perspective on Model Compositionality and Incremental Learningcited by 18
  11. 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
  12. 2024ATM: Improving Model Merging by Alternating Tuning and Mergingcited by 10
11 more
  1. 2024Harmony in Diversity: Merging Neural Networks with Canonical Correlation Analysiscited by 9
  2. 2025PEER pressure: Model-to-Model Regularization for Single Source Domain Generalizationcited by 7
  3. 2025On Task Vectors and Gradientscited by 7
  4. 2025Balanced Direction from Multifarious Choices: Arithmetic Meta-Learning for Domain Generalizationcited by 5
  5. 2025FCOS: A Two-Stage Recoverable Model Pruning Framework for Automatic Modulation Recognitioncited by 4
  6. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 4
  7. 2025Model Parallelism With Subnetwork Data Parallelismcited by 3
  8. 2024Fusion-PSRO: Nash Policy Fusion for Policy Space Response Oraclescited by 2
  9. 2023Do the Frankenstein, or how to achieve better out-of-distribution performance with manifold mixing model soupcited by 1
  10. 2024Non-Uniform Parameter-Wise Model Mergingcited by 0
  11. 2025Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learningcited by 0