mlsanityData to 7 Oct 2026

Linear Mode Connectivity and the Lottery Ticket Hypothesis

arXiv 1912.05671 ↗First posted 11 Dec 2019cs.LG cs.NE stat.MLcited by 627

Not on the map. Why?

Abstract (arXiv)

We study whether a neural network optimizes to the same, linearly connected minimum under different samples of SGD noise (e.g., random data order and augmentation). We find that standard vision models become stable to SGD noise in this way early in training. From then on, the outcome of optimization is determined to a linearly connected region. We use this technique to study iterative magnitude pruning (IMP), the procedure used by work on the lottery ticket hypothesis to identify subnetworks that could have trained in isolation to full accuracy. We find that these subnetworks only reach full accuracy when they are stable to SGD noise, which either occurs at initialization for small-scale settings (MNIST) or early in training for large-scale settings (ResNet-50 and Inception-v3 on ImageNet).

Key facts. Cited by 627 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 63; 2022: 84; 2023: 123; 2024: 135; 2025: 123; 2026: 99.

627 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  3. 2022Editing Models with Task Arithmeticcited by 1,076
  4. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  5. 2021Robust fine-tuning of zero-shot modelscited by 785
  6. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  7. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  8. 2023LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPScited by 420
  9. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  10. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  11. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  12. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
38 more
  1. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  2. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  3. 2023Model Sparsity Can Simplify Machine Unlearningcited by 206
  4. 2021Revisiting Model Stitching to Compare Neural Representationscited by 203
  5. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  6. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  7. 2022Patching open-vocabulary models by interpolating weightscited by 187
  8. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  9. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  10. 2023Understanding plasticity in neural networkscited by 170
  11. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  12. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuningcited by 167
  13. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  14. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  15. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  16. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  17. 2024Towards Safer Large Language Models through Machine Unlearningcited by 134
  18. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  19. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  20. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  21. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  22. 2021A Unified Lottery Ticket Hypothesis for Graph Neural Networkscited by 118
  23. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  24. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  25. 2022Fusing finetuned models for better pretrainingcited by 110
  26. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  27. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
  28. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  29. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
  30. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  31. 2023Deep Model Fusion: A Surveycited by 89
  32. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  33. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  34. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  35. 2021Learning Neural Network Subspacescited by 82
  36. 2023Modular Deep Learningcited by 82
  37. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  38. 2022When Do Flat Minima Optimizers Work?cited by 78

50 most cited of 627 All 627 (JSON)