# Essentially No Barriers in Neural Network Energy Landscape

arXiv: https://arxiv.org/abs/1803.00885 · HTML: https://mlsanity.com/paper/1803.00885/

## Key facts

- Cited by 330 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2018.

## Built on by (most cited first, 30 of 330)

- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [Revisiting Model Stitching to Compare Neural Representations](https://mlsanity.com/paper/2106.07682/) (2021)
- [ZipIt! Merging Models from Different Tasks without Training](https://mlsanity.com/paper/2305.03053/) (2023)
- [Diverse Weight Averaging for Out-of-Distribution Generalization](https://mlsanity.com/paper/2205.09739/) (2022)
- [Localizing Task Information for Improved Model Merging and Compression](https://mlsanity.com/paper/2405.07813/) (2024)
- [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021)
- [Model merging with SVD to tie the Knots](https://mlsanity.com/paper/2410.19735/) (2024)
- [REPAIR: REnormalizing Permuted Activations for Interpolation Repair](https://mlsanity.com/paper/2211.08403/) (2022)
- [Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging](https://mlsanity.com/paper/2406.15479/) (2024)
- [Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy](https://mlsanity.com/paper/2310.01334/) (2023)
- [What Happens after SGD Reaches Zero Loss? --A Mathematical Framework](https://mlsanity.com/paper/2110.06914/) (2021)
- [Improving Generalization in Federated Learning by Seeking Flat Minima](https://mlsanity.com/paper/2203.11834/) (2022)
- [Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization](https://mlsanity.com/paper/2212.10445/) (2022)
- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [Merging Multi-Task Models via Weight-Ensembling Mixture of Experts](https://mlsanity.com/paper/2402.00433/) (2024)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [Learning Neural Network Subspaces](https://mlsanity.com/paper/2102.10472/) (2021)
- [When Do Flat Minima Optimizers Work?](https://mlsanity.com/paper/2202.00661/) (2022)
- [Rehearsal revealed: The limits and merits of revisiting samples in continual learning](https://mlsanity.com/paper/2104.07446/) (2021)
- [Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling](https://mlsanity.com/paper/2102.13042/) (2021)
- [Re-basin via implicit Sinkhorn differentiation](https://mlsanity.com/paper/2212.12042/) (2022)
- [Revisiting Weighted Aggregation in Federated Learning with Neural Networks](https://mlsanity.com/paper/2302.10911/) (2023)
- [Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective](https://mlsanity.com/paper/2110.03095/) (2021)
- [Linear Connectivity Reveals Generalization Strategies](https://mlsanity.com/paper/2205.12411/) (2022)

## Abstract (arXiv)

Training neural networks involves finding minima of a high-dimensional non-convex loss function. Knowledge of the structure of this energy landscape is sparse. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that neural networks have enough capacity for structural changes, or that these changes are small between minima. Also, each minimum has at least one vanishing Hessian eigenvalue in addition to those resulting from trivial invariance.