mlsanityData to 7 Oct 2026

Visualizing the Loss Landscape of Neural Nets

arXiv 1712.09913 ↗First posted 28 Dec 2017cs.LG cs.CV stat.MLcited by 1,126

Not on the map. Why?

Abstract (arXiv)

Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effects on the underlying loss landscape, are not well understood. In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple "filter normalization" method that helps us visualize loss function curvature and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture affects the loss landscape, and how training parameters affect the shape of minimizers.

Key facts. Cited by 1,126 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 144; 2022: 202; 2023: 192; 2024: 225; 2025: 195; 2026: 167.

1,126 papers.

Sort:

By area:

  1. 2022Editing Models with Task Arithmeticcited by 1,076
  2. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  3. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 409
  4. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 376
  5. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  6. 2022How Do Vision Transformers Work?cited by 305
  7. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  8. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  9. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  10. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  11. 2021Revisiting Model Stitching to Compare Neural Representationscited by 203
  12. 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 192
38 more
  1. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  2. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  3. 2023Understanding plasticity in neural networkscited by 170
  4. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  5. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  6. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  7. 2022Reconstructing Training Data with Informed Adversariescited by 143
  8. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  9. 2023Do We Really Need Complicated Model Architectures For Temporal Networks?cited by 141
  10. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  11. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  12. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  13. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 121
  14. 2022FedALA: Adaptive Local Aggregation for Personalized Federated Learningcited by 121
  15. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  16. 2022A Comprehensive Survey on Trustworthy Graph Neural Networks: Privacy, Robustness, Fairness, and Explainabilitycited by 118
  17. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  18. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  19. 2024A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networkscited by 103
  20. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  21. 2021Representational Continuity for Unsupervised Continual Learningcited by 95
  22. 2022Sharpness-Aware Training for Freecited by 95
  23. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  24. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  25. 2023Deep Model Fusion: A Surveycited by 89
  26. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attackscited by 88
  27. 2024Model Stock: All we need is just a few fine-tuned modelscited by 87
  28. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  29. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  30. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  31. 2021Learning Neural Network Subspacescited by 82
  32. 2022When Do Flat Minima Optimizers Work?cited by 78
  33. 2021When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?cited by 76
  34. 2021Advancing Spiking Neural Networks towards Deep Residual Learningcited by 76
  35. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  36. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  37. 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 72
  38. 2021Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learningcited by 71

50 most cited of 1,126 All 1,126 (JSON)