mlsanityData to 7 Oct 2026

Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity

arXiv 2307.08286 ↗First posted 17 Jul 2023cs.LG cs.AI cs.CVcited by 50

Not on the map. Why?

Abstract (arXiv)

Recent work has revealed many intriguing empirical phenomena in neural network training, despite the poorly understood and highly complex loss landscapes and training dynamics. One of these phenomena, Linear Mode Connectivity (LMC), has gained considerable attention due to the intriguing observation that different solutions can be connected by a linear path in the parameter space while maintaining near-constant training and test losses. In this work, we introduce a stronger notion of linear connectivity, Layerwise Linear Feature Connectivity (LLFC), which says that the feature maps of every layer in different trained networks are also linearly connected. We provide comprehensive empirical evidence for LLFC across a wide range of settings, demonstrating that whenever two trained networks satisfy LMC (via either spawning or permutation methods), they also satisfy LLFC in nearly all the layers. Furthermore, we delve deeper into the underlying factors contributing to LLFC, which reveal new insights into the spawning and permutation approaches. The study of LLFC transcends and advances our understanding of LMC by adopting a feature-learning perspective.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

27 of 39 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  4. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  5. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  6. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  7. 2020Federated Learning with Matched Averagingcited by 564
  8. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  9. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  10. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  11. 2014Understanding image representations by measuring their equivariance and equivalencecited by 240
  12. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
15 more
  1. 2019Model Fusion via Optimal Transportcited by 217
  2. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  3. 2021Revisiting Model Stitching to Compare Neural Representationscited by 203
  4. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  5. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  6. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  7. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 85
  8. 2019Neural Graph Matching Network: Learning Lawler's Quadratic Assignment Problem with Extension to Hypergraph and Multiple-graph Matchingcited by 83
  9. 2019Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Netscited by 63
  10. 2019On Connected Sublevel Sets in Deep Learningcited by 51
  11. 2022Mechanistic Mode Connectivitycited by 50
  12. 2018On the loss landscape of a class of deep neural networks with no bad local valleyscited by 38
  13. 2018Understanding the Loss Surface of Neural Networks for Binary Classificationcited by 27
  14. 2021Scaling Up Exact Neural Network Compression by ReLU Stabilitycited by 19
  15. 2023Benign Overfitting in Deep Neural Networks under Lazy Trainingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 15; 2025: 24; 2026: 8.

50 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2024Parameter Competition Balancing for Model Mergingcited by 71
  3. 2024On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigmcited by 25
  4. 2023Proving Linear Mode Connectivity of Neural Networks via Optimal Transportcited by 24
  5. 2024C2M3: Cycle-Consistent Multi-Model Mergingcited by 20
  6. 2024DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptationcited by 19
  7. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  8. 2025Generalized Linear Mode Connectivity for Transformerscited by 14
  9. 2023Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theorycited by 12
  10. 2025From Task-Specific Models to Unified Systems: A Review of Model Merging Approachescited by 11
  11. 2025Understanding Mode Connectivity via Parameter Space Symmetrycited by 10
  12. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 9
38 more
  1. 2025The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditionscited by 9
  2. 2023Disentangling Linear Mode-Connectivitycited by 8
  3. 2024Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methodscited by 8
  4. 2025On Linear Mode Connectivity of Mixture-of-Experts Architecturescited by 8
  5. 2024Landscaping Linear Mode Connectivitycited by 7
  6. 2024A LoRA is Worth a Thousand Picturescited by 7
  7. 2024Training-free Heterogeneous Model Mergingcited by 5
  8. 2025Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptationcited by 5
  9. 2024Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Taskscited by 4
  10. 2024Vanishing Feature: Diagnosing Model Merging and Beyondcited by 3
  11. 2025Connecting Independently Trained Modes via Layer-Wise Connectivitycited by 3
  12. 2025New Evidence of the Two-Phase Learning Dynamics of Neural Networkscited by 3
  13. 2025Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalizationcited by 2
  14. 2025Analyzing the Role of Permutation Invariance in Linear Mode Connectivitycited by 2
  15. 2025On the Cone Effect in the Learning Dynamicscited by 2
  16. 2025Merge Now, Regret Later: The Hidden Cost of Model Merging Is Adversarial Transferabilitycited by 2
  17. 2025Do We Really Need Permutations? Impact of Model Width on Linear Mode Connectivitycited by 2
  18. 2026M-Loss: Quantifying Model Merging Compatibility with Limited Unlabeled Datacited by 2
  19. 2026Alignment Dynamics in LLM Fine-Tuningcited by 2
  20. 2026Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformerscited by 2
  21. 2024Linear Mode Connectivity in Differentiable Tree Ensemblescited by 1
  22. 2024CopRA: A Progressive LoRA Training Strategycited by 1
  23. 2025SE-Merging: A Self-Enhanced Approach for Dynamic Model Mergingcited by 1
  24. 2025Robustness and Regularization in Hierarchical Re-Basincited by 1
  25. 2025Exploring and Leveraging Class Vectors for Classifier Editingcited by 1
  26. 2025Weight Weaving: Parameter Pooling for Data-Free Model Mergingcited by 1
  27. 2025Linear Mode Connectivity under Data Shifts for Deep Ensembles of Image Classifierscited by 1
  28. 2026Model Agreement via Anchoringcited by 1
  29. 2026From Parameters to Feature Space: Task Arithmetic for Backdoor Mitigation in Model Mergingcited by 1
  30. 2024Manifold Metric: A Loss Landscape Approach for Predicting Model Performancecited by 0
  31. 2024Deep Model Merging: The Sister of Neural Network Interpretability -- A Surveycited by 0
  32. 2025MoMa: A Modular Deep Learning Framework for Material Property Predictioncited by 0
  33. 2025Finding Stable Subnetworks at Initialization with Dataset Distillationcited by 0
  34. 2025FroM: Frobenius Norm-Based Data-Free Adaptive Model Mergingcited by 0
  35. 2025Entropic Confinement and Mode Connectivity in Overparameterized Neural Networkscited by 0
  36. 2026Communication-Efficient Personalized Adaptation via Federated-Local Model Mergingcited by 0
  37. 2026Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiabilitycited by 0
  38. 2026Misalignment of Low-Loss Regions Causes Grokkingcited by 0