mlsanityData to 7 Oct 2026

Learning Neural Network Subspaces

arXiv 2102.10472 ↗First posted 20 Feb 2021cs.LG cs.CVcited by 82

Not on the map. Why?

Abstract (arXiv)

Recent observations have advanced our understanding of the neural network optimization landscape, revealing the existence of (1) paths of high accuracy containing diverse solutions and (2) wider minima offering improved performance. Previous methods observing diverse paths require multiple training runs. In contrast we aim to leverage both property (1) and (2) with a single method and in a single training run. With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks. These neural network subspaces contain diverse solutions that can be ensembled, approaching the ensemble performance of independently trained networks without the training cost. Moreover, using the subspace midpoint boosts accuracy, calibration, and robustness to label noise, outperforming Stochastic Weight Averaging.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Builds on 39 of its 53 references that are in the map. First posted on arXiv 20 Feb 2021.

Builds on

39 of 53 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Layer Normalizationcited by 5,579
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  9. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  10. 2016Wide Residual Networkscited by 3,057
  11. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  12. 2018Group Normalizationcited by 1,514
27 more
  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  2. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  4. 2019When Does Label Smoothing Help?cited by 985
  5. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  6. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  7. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  8. 2015Weight Uncertainty in Neural Networkscited by 554
  9. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  10. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  11. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  12. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  13. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  14. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  15. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  16. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  17. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  18. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  19. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  20. 2015MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemscited by 195
  21. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  22. 2019Exploring Randomly Wired Neural Networks for Image Recognitioncited by 103
  23. 2019Subspace Inference for Bayesian Deep Learningcited by 94
  24. 2019Large Scale Structure of Neural Network Loss Landscapescited by 68
  25. 2019The Difficulty of Training Sparse Neural Networkscited by 62
  26. 2020Neural networks with late-phase weightscited by 29
  27. 2020Revisiting "Qualitatively Characterizing Neural Network Optimization Problems"cited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 20; 2023: 17; 2024: 19; 2025: 13; 2026: 6.

82 papers.

Sort:

By area:

  1. 2022Editing Models with Task Arithmeticcited by 1,076
  2. 2021Robust fine-tuning of zero-shot modelscited by 785
  3. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  4. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  5. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  6. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  7. 2023Deep Model Fusion: A Surveycited by 89
  8. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  9. 2022Diversify and Disambiguate: Learning From Underspecified Datacited by 55
  10. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  11. 2022Mechanistic Mode Connectivitycited by 50
  12. 2022CoSCL: Cooperation of Small Continual Learners is Stronger than a Big Onecited by 43
38 more
  1. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 42
  2. 2024Towards Scalable and Versatile Weight Space Learningcited by 40
  3. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39
  4. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 38
  5. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  6. 2021Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connectorcited by 34
  7. 2022Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task modelscited by 29
  8. 2022Building a Subspace of Policies for Scalable Continual Learningcited by 28
  9. 2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 26
  10. 2024Asynchronous Local-SGD Training for Language Modelingcited by 24
  11. 2023Layer-wise Linear Mode Connectivitycited by 22
  12. 2023DART: Diversify-Aggregate-Repeat Training Improves Generalization of Neural Networkscited by 20
  13. 2022lo-fi: distributed fine-tuning without communicationcited by 19
  14. 2022Continual Learning Beyond a Single Modelcited by 18
  15. 2021Learning a subspace of policies for online adaptation in Reinforcement Learningcited by 14
  16. 2023Improving Online Continual Learning Performance and Stability with Temporal Ensemblescited by 14
  17. 2024QT-DoG: Quantization-aware Training for Domain Generalizationcited by 13
  18. 2021Connecting Low-Loss Subspace for Personalized Federated Learningcited by 12
  19. 2025TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillationcited by 12
  20. 2023MIST: Defending Against Membership Inference Attacks Through Membership-Invariant Subspace Trainingcited by 11
  21. 2023Revisiting Deep Ensemble for Out-of-Distribution Detection: A Loss Landscape Perspectivecited by 10
  22. 2024Simultaneous linear connectivity of neural networks modulo permutationcited by 9
  23. 2025The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditionscited by 9
  24. 2024Collective Model Intelligence Requires Compatible Specializationcited by 7
  25. 2022Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Trainingcited by 6
  26. 2022Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbationcited by 6
  27. 2022Learn2Weight: Parameter Adaptation against Similar-domain Adversarial Attackscited by 5
  28. 2024Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learningcited by 5
  29. 2025CENSOR: Defense Against Gradient Inversion via Orthogonal Subspace Bayesian Samplingcited by 5
  30. 2022Low-Loss Subspace Compression for Clean Gains against Multi-Agent Backdoor Attackscited by 4
  31. 2023Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Modelscited by 4
  32. 2024How Good is a Single Basin?cited by 4
  33. 2024Glider: Global and Local Instruction-Driven Expert Routercited by 4
  34. 2021LCS: Learning Compressible Subspaces for Adaptive Network Compression at Inference Timecited by 3
  35. 2022Interpolating Compressed Parameter Subspacescited by 3
  36. 2023Slimmable Encoders for Flexible Split DNNs in Bandwidth and Resource Constrained IoT Systemscited by 3
  37. 2024Bayesian Semi-structured Subspace Inferencecited by 3
  38. 2024Federated Learning over Connected Modescited by 3

50 most cited of 82 All 82 (JSON)