mlsanityData to 14 Oct 2026

Essentially No Barriers in Neural Network Energy Landscape

arXiv 1803.00885 ↗First posted 2 Mar 2018stat.ML cs.AI cs.LGcited by 330

Not on the map. Why?

Abstract (arXiv)

Training neural networks involves finding minima of a high-dimensional non-convex loss function. Knowledge of the structure of this energy landscape is sparse. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that neural networks have enough capacity for structural changes, or that these changes are small between minima. Also, each minimum has at least one vanishing Hessian eigenvalue in addition to those resulting from trivial invariance.

Key facts. Cited by 330 papers in the map (data through 14 Oct 2026). First posted on arXiv 2 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 35; 2022: 44; 2023: 57; 2024: 52; 2025: 60; 2026: 82.

330 papers.

Sort:

By area:

  1. 2022Editing Models with Task Arithmeticcited by 1,120
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  3. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  4. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  5. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
  6. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  7. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  8. 2021Revisiting Model Stitching to Compare Neural Representationscited by 209
  9. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  10. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  11. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
  12. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
38 more
  1. 2024Model merging with SVD to tie the Knotscited by 122
  2. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 120
  3. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 116
  4. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  5. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  6. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  7. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  8. 2023Deep Model Fusion: A Surveycited by 91
  9. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  10. 2023Modular Deep Learningcited by 83
  11. 2021Learning Neural Network Subspacescited by 82
  12. 2022When Do Flat Minima Optimizers Work?cited by 78
  13. 2021Rehearsal revealed: The limits and merits of revisiting samples in continual learningcited by 76
  14. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  15. 2022Re-basin via implicit Sinkhorn differentiationcited by 62
  16. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 58
  17. 2021Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspectivecited by 57
  18. 2022Linear Connectivity Reveals Generalization Strategiescited by 56
  19. 2025Deep Learning is Not So Mysterious or Differentcited by 56
  20. 2024What Matters for Model Merging at Scale?cited by 55
  21. 2022Mechanistic Mode Connectivitycited by 50
  22. 2023Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivitycited by 50
  23. 2022Anticorrelated Noise Injection for Improved Generalizationcited by 42
  24. 2023Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learningcited by 41
  25. 2024Towards Scalable and Versatile Weight Space Learningcited by 41
  26. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 41
  27. 2023π-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolationcited by 40
  28. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  29. 2023Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusioncited by 37
  30. 2023Revisiting the Noise Model of Stochastic Gradient Descentcited by 36
  31. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  32. 2025Symmetry in Neural Network Parameter Spacescited by 36
  33. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  34. 2023Equivariant Deep Weight Space Alignmentcited by 35
  35. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  36. 2021Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connectorcited by 34
  37. 2022Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task modelscited by 29
  38. 2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 28

50 most cited of 330 All 330 (JSON)