mlsanityData to 7 Oct 2026

Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling

arXiv 2102.13042 ↗First posted 25 Feb 2021cs.LG cs.CV stat.MLcited by 75

Not on the map. Why?

Abstract (arXiv)

With a better understanding of the loss surfaces for multilayer networks, we can build more robust and accurate training procedures. Recently it was discovered that independently trained SGD solutions can be connected along one-dimensional paths of near-constant training loss. In this paper, we show that there are mode-connecting simplicial complexes that form multi-dimensional manifolds of low loss, connecting many independently trained models. Inspired by this discovery, we show how to efficiently build simplicial complexes for fast ensembling, outperforming independently trained deep ensembles in accuracy, calibration, and robustness to dataset shift. Notably, our approach only requires a few training epochs to discover a low-loss simplex, starting from a pre-trained solution. Code is available at https://github.com/g-benton/loss-surface-simplexes.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 25 Feb 2021.

Builds on

23 of 28 references.

References · 23

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  9. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  10. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  11. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  12. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
11 more
  1. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  2. 2014The Loss Surfaces of Multilayer Networkscited by 278
  3. 2014Qualitatively characterizing neural network optimization problemscited by 186
  4. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  5. 2019Subspace Inference for Bayesian Deep Learningcited by 94
  6. 2021Learning Neural Network Subspacescited by 82
  7. 2019Large Scale Structure of Neural Network Loss Landscapescited by 68
  8. 2019Understanding Generalization through Visualizationscited by 54
  9. 2020Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisitedcited by 50
  10. 2019Loss Patterns of Neural Networkscited by 10
  11. 2019A Deep Neural Network's Loss Surface Contains Every Low-dimensional Patterncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 11; 2023: 13; 2024: 22; 2025: 17; 2026: 6.

75 papers.

Sort:

By area:

  1. 2022Editing Models with Task Arithmeticcited by 1,076
  2. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  3. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  4. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  5. 2023Deep Model Fusion: A Surveycited by 89
  6. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  7. 2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
  8. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  9. 2021Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspectivecited by 56
  10. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  11. 2025Deep Learning is Not So Mysterious or Differentcited by 55
  12. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
38 more
  1. 2024What Matters for Model Merging at Scale?cited by 52
  2. 2022Mechanistic Mode Connectivitycited by 50
  3. 2024Towards Scalable and Versatile Weight Space Learningcited by 40
  4. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 38
  5. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  6. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  7. 2021Dangers of Bayesian Model Averaging under Covariate Shiftcited by 31
  8. 2025Symmetry in Neural Network Parameter Spacescited by 31
  9. 2022Building a Subspace of Policies for Scalable Continual Learningcited by 28
  10. 2022lo-fi: distributed fine-tuning without communicationcited by 19
  11. 2022Continual Learning Beyond a Single Modelcited by 18
  12. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  13. 2021Learning a subspace of policies for online adaptation in Reinforcement Learningcited by 14
  14. 2023Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusioncited by 13
  15. 2024QT-DoG: Quantization-aware Training for Domain Generalizationcited by 13
  16. 2021Connecting Low-Loss Subspace for Personalized Federated Learningcited by 12
  17. 2024Task Arithmetic Through The Lens Of One-Shot Federated Learningcited by 12
  18. 2025TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillationcited by 12
  19. 2024FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusioncited by 10
  20. 2025Understanding Mode Connectivity via Parameter Space Symmetrycited by 10
  21. 2024Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivitycited by 9
  22. 2024WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Averagecited by 9
  23. 2023Disentangling Linear Mode-Connectivitycited by 8
  24. 2024SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Modelscited by 8
  25. 2024Improving Model Fusion by Training-time Neuron Alignment with Fixed Neuron Anchorscited by 7
  26. 2023To Stay or Not to Stay in the Pre-train Basin: Insights on Ensembling in Transfer Learningcited by 6
  27. 2024Do Deep Neural Network Solutions Form a Star Domain?cited by 6
  28. 2025Paths and Ambient Spaces in Neural Loss Landscapescited by 6
  29. 2023Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Modelscited by 4
  30. 2023On permutation symmetries in Bayesian neural network posteriors: a variational perspectivecited by 4
  31. 2024How Good is a Single Basin?cited by 4
  32. 2024Subgraph Aggregation for Out-of-Distribution Generalization on Graphscited by 4
  33. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 4
  34. 2026Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizerscited by 4
  35. 2023From Charts to Atlas: Merging Latent Spaces into Onecited by 3
  36. 2024Federated Learning over Connected Modescited by 3
  37. 2025Estimating the Probability of Sampling a Trained Neural Network at Randomcited by 3
  38. 2025Connecting Independently Trained Modes via Layer-Wise Connectivitycited by 3

50 most cited of 75 All 75 (JSON)