mlsanityData to 7 Oct 2026

Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry

arXiv 2202.03038 ↗First posted 7 Feb 2022cs.LG cond-mat.dis-nncited by 27

Not on the map. Why?

Abstract (arXiv)

We systematize the approach to the investigation of deep neural network landscapes by basing it on the geometry of the space of implemented functions rather than the space of parameters. Grouping classifiers into equivalence classes, we develop a standardized parameterization in which all symmetries are removed, resulting in a toroidal topology. On this space, we explore the error landscape rather than the loss. This lets us derive a meaningful notion of the flatness of minimizers and of the geodesic paths connecting them. Using different optimization algorithms that sample minimizers with different flatness we study the mode connectivity and relative distances. Testing a variety of state-of-the-art architectures and benchmark datasets, we confirm the correlation between flatness and generalization performance; we further show that in function space flatter minima are closer to each other and that the barriers along the geodesics connecting them are small. We also find that minimizers found by variants of gradient descent can be connected by zero-error paths composed of two straight lines in parameter space, i.e. polygonal chains with a single bend. We observe similar qualitative results in neural networks with binary weights and activations, providing one of the first results concerning the connectivity in this setting. Our results hinge on symmetry removal, and are in remarkable agreement with the rich phenomenology described by some recent analytical studies performed on simple shallow models.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 7 Feb 2022.

Builds on

32 of 45 references.

References · 32

  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  5. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  6. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  7. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  8. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  9. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  10. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  11. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  12. 2019Model Fusion via Optimal Transportcited by 217
20 more
  1. 2020Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustnesscited by 130
  2. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  3. 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 125
  4. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  5. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  6. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 85
  7. 2019Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Netscited by 63
  8. 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 61
  9. 2019Bad Global Minima Exist and SGD Can Reach Themcited by 49
  10. 2019Shaping the learning landscape in neural networks around wide flat minimacited by 47
  11. 2016Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemescited by 40
  12. 2015Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapsescited by 39
  13. 2019Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networkscited by 35
  14. 2021Unveiling the structure of wide flat minima in neural networkscited by 27
  15. 2020Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized modelscited by 26
  16. 2020Entropic gradient descent algorithms and wide flat minimacited by 25
  17. 2019Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activationscited by 22
  18. 2021Learning through atypical "phase transitions" in overparameterized neural networkscited by 21
  19. 2018Using Mode Connectivity for Loss Landscape Analysiscited by 19
  20. 2017On the role of synaptic stochasticity in training low-precision neural networkscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 5; 2024: 6; 2025: 6; 2026: 5.

27 papers.

Sort:

By area:

  1. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  2. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  3. 2023Deep Model Fusion: A Surveycited by 89
  4. 2022Mechanistic Mode Connectivitycited by 50
  5. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  6. 2025Symmetry in Neural Network Parameter Spacescited by 31
  7. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  8. 2023Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetrycited by 20
  9. 2023Typical and atypical solutions in non-convex neural networks with discrete and continuous weightscited by 13
  10. 2025Understanding Mode Connectivity via Parameter Space Symmetrycited by 10
  11. 2022Boundary between noise and information applied to filtering neural network weight matricescited by 9
  12. 2024Topological obstruction to the training of shallow ReLU neural networkscited by 6
15 more
  1. 2025Paths and Ambient Spaces in Neural Loss Landscapescited by 6
  2. 2024FlatNAS: optimizing Flatness in Neural Architecture Search for Out-of-Distribution Robustnesscited by 5
  3. 2023High-dimensional manifold of solutions in neural networks: insights from statistical physicscited by 4
  4. 2023On permutation symmetries in Bayesian neural network posteriors: a variational perspectivecited by 4
  5. 2024Vanishing Feature: Diagnosing Model Merging and Beyondcited by 3
  6. 2026DeepWeightFlow: Re-Basined Flow Matching for Generating Neural Network Weightscited by 3
  7. 2024Weight Scope Alignment: A Frustratingly Easy Method for Model Mergingcited by 2
  8. 2025Architecture-Aware Minimization (A2M): How to Find Flat Minima in Neural Architecture Searchcited by 2
  9. 2026On the Epistemic Uncertainty of Overparametrized Neural Networkscited by 2
  10. 2024Training Multi-Layer Binary Neural Networks With Local Binary Error Signalscited by 1
  11. 2025The Empirical Impact of Reducing Symmetries on the Performance of Deep Ensembles and MoEcited by 1
  12. 2025Symmetry-Aware Graph Metanetwork Autoencoders: Model Merging through Parameter Canonicalizationcited by 0
  13. 2026HERCULES: Hardware-Efficient, Robust, Continual Learning Neural Architecture Searchcited by 0
  14. 2026Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiabilitycited by 0
  15. 2026When a Flatness Proxy Is Not a Function: Robustness Certificates and Training Interventionscited by 0