mlsanityData to 7 Oct 2026

Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes

arXiv 2104.11044 ↗First posted 22 Apr 2021cs.LG cs.AI stat.MLcited by 27

Not on the map. Why?

Abstract (arXiv)

Linear interpolation between initial neural network parameters and converged parameters after training with stochastic gradient descent (SGD) typically leads to a monotonic decrease in the training objective. This Monotonic Linear Interpolation (MLI) property, first observed by Goodfellow et al. (2014) persists in spite of the non-convex objectives and highly non-linear training dynamics of neural networks. Extending this work, we evaluate several hypotheses for this property that, to our knowledge, have not yet been explored. Using tools from differential geometry, we draw connections between the interpolated paths in function space and the monotonicity of the network - providing sufficient conditions for the MLI property under mean squared error. While the MLI property holds under various settings (e.g. network architectures and learning problems), we show in practice that networks violating the MLI property can be produced systematically, by encouraging the weights to move far from initialization. The MLI property raises important questions about the loss landscape geometry of neural networks and highlights the need to further study their global properties.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 47 of its 62 references that are in the map. First posted on arXiv 22 Apr 2021.

Builds on

47 of 62 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  11. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  12. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
35 more
  1. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  5. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  6. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  7. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  8. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  9. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  10. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  11. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  12. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  13. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  14. 2014Qualitatively characterizing neural network optimization problemscited by 186
  15. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  16. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  17. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  18. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  19. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 116
  20. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  21. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  22. 2019Optimization for deep learning: theory and algorithmscited by 70
  23. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 68
  24. 2019Large Scale Structure of Neural Network Loss Landscapescited by 68
  25. 2019Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Netscited by 63
  26. 2019On Connected Sublevel Sets in Deep Learningcited by 51
  27. 2019Emergent properties of the local geometry of neural loss landscapescited by 46
  28. 2019Loss Landscapes of Regularized Linear Autoencoderscited by 45
  29. 2020Disentangling Adaptive Gradient Methods from Learning Ratescited by 39
  30. 2019Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networkscited by 35
  31. 2018The Goldilocks zone: Towards better understanding of neural network loss landscapescited by 32
  32. 2020When Does Preconditioning Help or Hurt Generalization?cited by 30
  33. 2020Revisiting "Qualitatively Characterizing Neural Network Optimization Problems"cited by 22
  34. 2016An empirical analysis of the optimization of deep network loss surfacescited by 21
  35. 2018Spurious Valleys in Two-layer Neural Network Optimization Landscapescited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 8; 2023: 5; 2024: 5; 2025: 2; 2026: 3.

27 papers.

Sort:

By area:

  1. 2022Editing Models with Task Arithmeticcited by 1,076
  2. 2021Robust fine-tuning of zero-shot modelscited by 785
  3. 2022Fusing finetuned models for better pretrainingcited by 110
  4. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 38
  5. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  6. 2021What can linear interpolation of neural network loss landscapes tell us?cited by 27
  7. 2023Proving Linear Mode Connectivity of Neural Networks via Optimal Transportcited by 24
  8. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  9. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  10. 2024A Second-Order Perspective on Model Compositionality and Incremental Learningcited by 18
  11. 2021Towards Noise-adaptive, Problem-adaptive (Accelerated) Stochastic Gradient Descentcited by 16
  12. 2022Special Properties of Gradient Descent with Large Learning Ratescited by 10
15 more
  1. 2022Plateau in Monotonic Linear Interpolation -- A "Biased" View of Loss Landscape for Deep Networkscited by 10
  2. 2023Edit at your own risk: evaluating the robustness of edited models to distribution shiftscited by 8
  3. 2025On Linear Mode Connectivity of Mixture-of-Experts Architecturescited by 8
  4. 2022Linear Interpolation In Parameter Space is Good Enough for Fine-Tuned Language Modelscited by 7
  5. 2023No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Pathscited by 7
  6. 2026EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KLcited by 4
  7. 2022FuNNscope: Visual microscope for interactively exploring the loss landscape of fully connected neural networkscited by 2
  8. 2026Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Trainingcited by 2
  9. 2022Class Interference of Deep Neural Networkscited by 1
  10. 2024Visualizing, Rethinking, and Mining the Loss Landscape of Deep Neural Networkscited by 1
  11. 2026Visualizing the loss landscapes of physics-informed neural networkscited by 1
  12. 2021On Accelerating Distributed Convex Optimizationscited by 0
  13. 2024Hyper-Representations: Learning from Populations of Neural Networkscited by 0
  14. 2024In Search of the Successful Interpolation: On the Role of Sharpness in CLIP Generalizationcited by 0
  15. 2025Finetune Once: Decoupling General & Domain Learning with Dynamic Boosted Annealingcited by 0