mlsanityData to 7 Oct 2026

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

arXiv 1802.10026 ↗First posted 27 Feb 2018stat.ML cs.AI cs.LGcited by 585

Not on the map. Why?

Abstract (arXiv)

The loss functions of deep neural networks are complex and their geometric properties are not well understood. We show that the optima of these complex loss functions are in fact connected by simple curves over which training and test accuracy are nearly constant. We introduce a training procedure to discover these high-accuracy pathways between modes. Inspired by this new geometric insight, we also propose a new ensembling method entitled Fast Geometric Ensembling (FGE). Using FGE we can train high-performing ensembles in the time required to train a single model. We achieve improved performance compared to the recent state-of-the-art Snapshot Ensembles, on CIFAR-10, CIFAR-100, and ImageNet.

Key facts. Cited by 585 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 63; 2022: 86; 2023: 93; 2024: 124; 2025: 102; 2026: 117.

585 papers.

Sort:

By area:

  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2022Editing Models with Task Arithmeticcited by 1,076
  3. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  4. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  5. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  6. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  7. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  8. 2024The Platonic Representation Hypothesiscited by 362
  9. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  10. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  11. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  12. 2021Revisiting Model Stitching to Compare Neural Representationscited by 203
38 more
  1. 2021Fixing Data Augmentation to Improve Adversarial Robustnesscited by 201
  2. 2021Data Augmentation Can Improve Robustnesscited by 193
  3. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  4. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  5. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  6. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  7. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  8. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  9. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  10. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  11. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  12. 2024Model merging with SVD to tie the Knotscited by 118
  13. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  14. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  15. 2024TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensemblingcited by 105
  16. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  17. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  18. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  19. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  20. 2023Deep Model Fusion: A Surveycited by 89
  21. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  22. 2021Learning Neural Network Subspacescited by 82
  23. 2023Modular Deep Learningcited by 82
  24. 2022When Do Flat Minima Optimizers Work?cited by 78
  25. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  26. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 75
  27. 2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
  28. 2024Parameter Competition Balancing for Model Mergingcited by 71
  29. 2023Permutation Equivariant Neural Functionalscited by 69
  30. 2024Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilitiescited by 67
  31. 2021Representation Topology Divergence: A Method for Comparing Neural Network Representationscited by 66
  32. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 64
  33. 2022Re-basin via implicit Sinkhorn differentiationcited by 61
  34. 2024Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuningcited by 59
  35. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  36. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 57
  37. 2021Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspectivecited by 56
  38. 2022Linear Connectivity Reveals Generalization Strategiescited by 55

50 most cited of 585 All 585 (JSON)