# What can linear interpolation of neural network loss landscapes tell us?

arXiv: https://arxiv.org/abs/2106.16004 · HTML: https://mlsanity.com/paper/2106.16004/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 40 references that are in the map.
- First posted on arXiv 30 Jun 2021.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)
- [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018)
- [An Empirical Model of Large-Batch Training](https://mlsanity.com/paper/1812.06162/) (2018)
- [Three Factors Influencing Minima in SGD](https://mlsanity.com/paper/1711.04623/) (2017)
- [Qualitatively characterizing neural network optimization problems](https://mlsanity.com/paper/1412.6544/) (2014)
- [Maxout Networks](https://mlsanity.com/paper/1302.4389/) (2013)
- [The Break-Even Point on Optimization Trajectories of Deep Neural Networks](https://mlsanity.com/paper/2002.09572/) (2020)
- [Are All Layers Created Equal?](https://mlsanity.com/paper/1902.01996/) (2019)
- [Large Scale Structure of Neural Network Loss Landscapes](https://mlsanity.com/paper/1906.04724/) (2019)
- [Visualizing and Understanding the Effectiveness of BERT](https://mlsanity.com/paper/1908.05620/) (2019)
- [Bad Global Minima Exist and SGD Can Reach Them](https://mlsanity.com/paper/1906.02613/) (2019)
- [Deep Learning is Singular, and That's Good](https://mlsanity.com/paper/2010.11560/) (2020)
- [The intriguing role of module criticality in the generalization of deep networks](https://mlsanity.com/paper/1912.00528/) (2019)
- [Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes](https://mlsanity.com/paper/2104.11044/) (2021)
- [Revisiting "Qualitatively Characterizing Neural Network Optimization Problems"](https://mlsanity.com/paper/2012.06898/) (2020)

## Built on by (most cited first, 28 of 28)

- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Revisiting Weighted Aggregation in Federated Learning with Neural Networks](https://mlsanity.com/paper/2302.10911/) (2023)
- [Proving Linear Mode Connectivity of Neural Networks via Optimal Transport](https://mlsanity.com/paper/2310.19103/) (2023)
- [Layer-wise Linear Mode Connectivity](https://mlsanity.com/paper/2307.06966/) (2023)
- [The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof](https://mlsanity.com/paper/2405.20231/) (2024)
- [Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and width](https://mlsanity.com/paper/2302.12250/) (2023)
- [Optimizers Qualitatively Alter Solutions And We Should Leverage This](https://mlsanity.com/paper/2507.12224/) (2025)
- [Simultaneous linear connectivity of neural networks modulo permutation](https://mlsanity.com/paper/2404.06498/) (2024)
- [Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity](https://mlsanity.com/paper/2404.06391/) (2024)
- [The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions](https://mlsanity.com/paper/2506.13234/) (2025)
- [Disentangling Linear Mode-Connectivity](https://mlsanity.com/paper/2312.09832/) (2023)
- [Improving Model Fusion by Training-time Neuron Alignment with Fixed Neuron Anchors](https://mlsanity.com/paper/2402.01342/) (2024)
- [High-dimensional manifold of solutions in neural networks: insights from statistical physics](https://mlsanity.com/paper/2309.09240/) (2023)
- [FuNNscope: Visual microscope for interactively exploring the loss landscape of fully connected neural networks](https://mlsanity.com/paper/2204.04511/) (2022)
- [Transferring Learning Trajectories of Neural Networks](https://mlsanity.com/paper/2305.14122/) (2023)
- [Weight Scope Alignment: A Frustratingly Easy Method for Model Merging](https://mlsanity.com/paper/2408.12237/) (2024)
- [Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training](https://mlsanity.com/paper/2605.26484/) (2026)
- [Class Interference of Deep Neural Networks](https://mlsanity.com/paper/2211.01370/) (2022)
- [Distilled Pruning: Using Synthetic Data to Win the Lottery](https://mlsanity.com/paper/2307.03364/) (2023)
- [Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks](https://mlsanity.com/paper/2405.12489/) (2024)
- [Visualizing, Rethinking, and Mining the Loss Landscape of Deep Neural Networks](https://mlsanity.com/paper/2405.12493/) (2024)
- [In Search of the Successful Interpolation: On the Role of Sharpness in CLIP Generalization](https://mlsanity.com/paper/2410.16476/) (2024)
- [The Geometric Anatomy of Capability Acquisition in Transformers](https://mlsanity.com/paper/2602.15997/) (2026)
- [Lost in the Non-convex Loss Landscape: How to Fine-tune the Large Time Series Model?](https://mlsanity.com/paper/2606.08578/) (2026)
- [FedGuCci: Making Local Models More Connected in Landscape for Federated Learning](https://mlsanity.com/paper/2402.18949/) (2024)
- [A Closer Look at Personalized Fine-Tuning in Heterogeneous Federated Learning](https://mlsanity.com/paper/2511.12695/) (2025)
- [Scalar Representations of Neural Network Training Dynamics](https://mlsanity.com/paper/2606.30384/) (2026)
- [Task Vector Descent: Learning from Non-IID Batches](https://mlsanity.com/paper/2610.05402/) (2026)

## Abstract (arXiv)

Studying neural network loss landscapes provides insights into the nature of the underlying optimization problems. Unfortunately, loss landscapes are notoriously difficult to visualize in a human-comprehensible fashion. One common way to address this problem is to plot linear slices of the landscape, for example from the initial state of the network to the final state after optimization. On the basis of this analysis, prior work has drawn broader conclusions about the difficulty of the optimization problem. In this paper, we put inferences of this kind to the test, systematically evaluating how linear interpolation and final performance vary when altering the data, choice of initialization, and other optimizer and architecture design choices. Further, we use linear interpolation to study the role played by individual layers and substructures of the network. We find that certain layers are more sensitive to the choice of initialization, but that the shape of the linear path is not indicative of the changes in test accuracy of the model. Our results cast doubt on the broader intuition that the presence or absence of barriers when interpolating necessarily relates to the success of optimization.