mlsanityData to 7 Oct 2026

Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective

arXiv 2203.08124 ↗First posted 15 Mar 2022cs.LG cs.CVcited by 57

Not on the map. Why?

Abstract (arXiv)

We discuss methods for visualizing neural network decision boundaries and decision regions. We use these visualizations to investigate issues related to reproducibility and generalization in neural network training. We observe that changes in model architecture (and its associate inductive bias) cause visible changes in decision boundaries, while multiple runs with the same architecture yield results with strong similarities, especially in the case of wide architectures. We also use decision boundary methods to visualize double descent phenomena. We see that decision boundary reproducibility depends strongly on model width. Near the threshold of interpolation, neural network decision boundaries become fragmented into many small decision regions, and these regions are non-reproducible. Meanwhile, very narrows and very wide networks have high levels of reproducibility in their decision boundaries with relatively few decision regions. We discuss how our observations relate to the theory of double descent phenomena in convex models. Code is available at https://github.com/somepago/dbViz

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

28 of 40 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  8. 2016Wide Residual Networkscited by 3,057
  9. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  10. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  11. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  12. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
16 more
  1. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  2. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  3. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  4. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  5. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  6. 2021Does Knowledge Distillation Really Work?cited by 187
  7. 2019Adversarially Robust Distillationcited by 124
  8. 2019Harmless interpolation of noisy data in regressioncited by 121
  9. 2018Are adversarial examples inevitable?cited by 113
  10. 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
  11. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  12. 2020Understanding Double Descent Requires a Fine-Grained Bias-Variance Decompositioncited by 75
  13. 2018A Modern Take on the Bias-Variance Tradeoff in Neural Networkscited by 64
  14. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  15. 2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 54
  16. 2019Characterizing the Decision Boundary of Deep Neural Networkscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 18; 2024: 9; 2025: 8; 2026: 8.

57 papers.

Sort:

By area:

  1. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  2. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  3. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  4. 2022Rashomon Capacity: A Metric for Predictive Multiplicity in Classificationcited by 46
  5. 2023Latent Space Translation via Semantic Alignmentcited by 46
  6. 2022Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queriescited by 36
  7. 2023SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundariescited by 31
  8. 2022What Knowledge Gets Distilled in Knowledge Distillation?cited by 26
  9. 2022Random initialisations performing above chance and how to find themcited by 23
  10. 2023The Emergence of Reproducibility and Generalizability in Diffusion Modelscited by 21
  11. 2023On the Variance of Neural Network Training with respect to Test Sets and Distributionscited by 20
  12. 2023Simplifying Neural Network Training Under Class Imbalancecited by 18
38 more
  1. 2025Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learningcited by 17
  2. 2022Towards Good Practices in Evaluating Transfer Adversarial Attackscited by 14
  3. 2023One-bit Flip is All You Need: When Bit-flip Attack Meets Model Trainingcited by 14
  4. 2023On the Lipschitz Constant of Deep Networks and Double Descentcited by 11
  5. 2024Latent Space Translation via Inverse Relative Projectioncited by 10
  6. 2025Towards Model Resistant to Transferable Adversarial Examples via Trigger Activationcited by 10
  7. 2022Task Discovery: Finding the Tasks that Neural Networks Generalize oncited by 9
  8. 2022A law of adversarial risk, interpolation, and label noisecited by 8
  9. 2022Deep Double Descent via Smooth Interpolationcited by 8
  10. 2023An Empirical Investigation into Benchmarking Model Multiplicity for Trustworthy Machine Learning: A Case Study on Image Classificationcited by 7
  11. 2023Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusioncited by 6
  12. 2023Reckoning with the Disagreement Problem: Explanation Consensus as a Training Objectivecited by 6
  13. 2023Unraveling the Enigma of Double Descent: An In-depth Analysis through the Lens of Learned Feature Spacecited by 6
  14. 2025Escaping Plato's Cave: Towards the Alignment of 3D and Text Latent Spacescited by 6
  15. 2023Deep neural networks architectures from the perspective of manifold learningcited by 5
  16. 2024Reprogramming Distillation for Medical Foundation Modelscited by 5
  17. 2022Similarity of Neural Architectures using Adversarial Attack Transferabilitycited by 4
  18. 2023Calibrating the Rigged Lottery: Making All Tickets Reliablecited by 4
  19. 2024The Cost of Arbitrariness for Individuals: Examining the Legal and Technical Challenges of Model Multiplicitycited by 4
  20. 2025Connecting Neural Models Latent Geometries with Relative Geodesic Representationscited by 4
  21. 2025When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspectivecited by 4
  22. 2024How Do the Architecture and Optimizer Affect Representation Learning? On the Training Dynamics of Representations in Deep Neural Networkscited by 2
  23. 2025Understanding Nonlinear Implicit Bias via Region Counts in Input Spacecited by 2
  24. 2022Frozen Overparameterization: A Double Descent Perspective on Transfer Learning of Deep Neural Networkscited by 1
  25. 2023The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNscited by 1
  26. 2023Robust Nonparametric Hypothesis Testing to Understand Variability in Training Neural Networkscited by 1
  27. 2023Efficient Key-Based Adversarial Defense for ImageNet by Using Pre-trained Modelcited by 1
  28. 2024Latent Functional Maps: a spectral framework for representation alignmentcited by 1
  29. 2025How Does Overparameterization Affect Machine Unlearning of Deep Neural Networks?cited by 1
  30. 2026Model Agreement via Anchoringcited by 1
  31. 2026Fingerprinting Deep Neural Networks for Ownership Protection: An Analytical Approachcited by 1
  32. 2026Unveiling Memorization-Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noisecited by 1
  33. 2022Effectiveness of Function Matching in Driving Scene Recognitioncited by 0
  34. 2022The Vanishing Decision Boundary Complexity and the Strong First Componentcited by 0
  35. 2023Perturbed examples reveal invariances shared by language modelscited by 0
  36. 2024A Personalized Video-Based Hand Taxonomy: Application for Individuals with Spinal Cord Injurycited by 0
  37. 2024Class-wise Activation Unravelling the Engima of Deep Double Descentcited by 0
  38. 2024Linking in Style: Understanding learned features in deep learning modelscited by 0

50 most cited of 57 All 57 (JSON)