mlsanityData to 7 Oct 2026

Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets

arXiv 1906.06247 ↗First posted 14 Jun 2019cs.LG stat.MLcited by 63

Not on the map. Why?

Abstract (arXiv)

Mode connectivity is a surprising phenomenon in the loss landscape of deep nets. Optima -- at least those discovered by gradient-based optimization -- turn out to be connected by simple paths on which the loss function is almost constant. Often, these paths can be chosen to be piece-wise linear, with as few as two segments. We give mathematical explanations for this phenomenon, assuming generic properties (such as dropout stability and noise stability) of well-trained deep nets, which have previously been identified as part of understanding the generalization properties of deep nets. Our explanation holds for realistic multilayer nets, and experiments are presented to verify the theory.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 13; 2023: 10; 2024: 12; 2025: 9; 2026: 5.

63 papers.

Sort:

By area:

  1. 2022Editing Models with Task Arithmeticcited by 1,076
  2. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  3. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  4. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  5. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  6. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  7. 2023Deep Model Fusion: A Surveycited by 89
  8. 2022Re-basin via implicit Sinkhorn differentiationcited by 61
  9. 2025Deep Learning is Not So Mysterious or Differentcited by 55
  10. 2024What Matters for Model Merging at Scale?cited by 52
  11. 2022Mechanistic Mode Connectivitycited by 50
  12. 2023Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivitycited by 50
38 more
  1. 2023Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learningcited by 41
  2. 2023Hidden symmetries of ReLU networkscited by 38
  3. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  4. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  5. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  6. 2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 27
  7. 2022Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometrycited by 27
  8. 2024On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigmcited by 25
  9. 2023Proving Linear Mode Connectivity of Neural Networks via Optimal Transportcited by 24
  10. 2024The Real Tropical Geometry of Neural Networkscited by 23
  11. 2022Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networkscited by 21
  12. 2021Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorizationcited by 15
  13. 2021Learning a subspace of policies for online adaptation in Reinforcement Learningcited by 14
  14. 2021Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networkscited by 13
  15. 2025TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillationcited by 12
  16. 2021When Are Solutions Connected in Deep Networks?cited by 10
  17. 2022Plateau in Monotonic Linear Interpolation -- A "Biased" View of Loss Landscape for Deep Networkscited by 10
  18. 2025Understanding Mode Connectivity via Parameter Space Symmetrycited by 10
  19. 2021Redundant representations help generalization in wide neural networkscited by 9
  20. 2022On Quantum Speedups for Nonconvex Optimization via Quantum Tunneling Walkscited by 9
  21. 2023Optimal Sets and Solution Paths of ReLU Networkscited by 9
  22. 2024Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivitycited by 9
  23. 2021A Note on Connectivity of Sublevel Sets in Deep Learningcited by 8
  24. 2022Deep Double Descent via Smooth Interpolationcited by 8
  25. 2024Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methodscited by 8
  26. 2024Landscaping Linear Mode Connectivitycited by 7
  27. 2024Do Deep Neural Network Solutions Form a Star Domain?cited by 6
  28. 2024Topological obstruction to the training of shallow ReLU neural networkscited by 6
  29. 2025On Learning Verifiers and Implications to Chain-of-Thought Reasoningcited by 6
  30. 2022Flatter, faster: scaling momentum for optimal speedup of SGDcited by 5
  31. 2023Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networkscited by 5
  32. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 4
  33. 2021Tighter Sparse Approximation Bounds for ReLU Neural Networkscited by 3
  34. 2022Mean-field analysis for heavy ball methods: Dropout-stability, connectivity, and global convergencecited by 3
  35. 2022Robust Federated Learning against both Data Heterogeneity and Poisoning Attack via Aggregation Optimizationcited by 3
  36. 2023Mode Connectivity in Auction Designcited by 3
  37. 2023Mode Connectivity and Data Heterogeneity of Federated Learningcited by 3
  38. 2024Activated Parameter Locating via Causal Intervention for Model Mergingcited by 3

50 most cited of 63 All 63 (JSON)