mlsanityData to 7 Oct 2026

The intriguing role of module criticality in the generalization of deep networks

arXiv 1912.00528 ↗First posted 2 Dec 2019cs.LG stat.MLcited by 40

Not on the map. Why?

Abstract (arXiv)

We study the phenomenon that some modules of deep neural networks (DNNs) are more critical than others. Meaning that rewinding their parameter values back to initialization, while keeping other modules fixed at the trained parameters, results in a large drop in the network's performance. Our analysis reveals interesting properties of the loss landscape which leads us to propose a complexity measure, called module criticality, based on the shape of the valleys that connects the initial and final values of the module parameters. We formulate how generalization relates to the module criticality, and show that this measure is able to explain the superior generalization performance of some architectures over others, whereas earlier measures fail to do so.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 9; 2023: 7; 2024: 10; 2025: 3; 2026: 2.

40 papers.

Sort:

By area:

  1. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  2. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  3. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  4. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  5. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  6. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  7. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  8. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  9. 2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
  10. 2021What can linear interpolation of neural network loss landscapes tell us?cited by 27
  11. 2023Improving Generalization of Adversarial Training via Robust Critical Fine-Tuningcited by 23
  12. 2023Layer-wise Linear Mode Connectivitycited by 22
28 more
  1. 2024Securely Fine-tuning Pre-trained Encoders Against Adversarial Examplescited by 21
  2. 2022Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classificationcited by 19
  3. 2022Limitations of Information-Theoretic Generalization Bounds for Gradient Descent Methods in Stochastic Convex Optimizationcited by 18
  4. 2023Layer-wise Auto-Weighting for Non-Stationary Test-Time Adaptationcited by 14
  5. 2023On the Lipschitz Constant of Deep Networks and Double Descentcited by 11
  6. 2021Disparity Between Batches as a Signal for Early Stoppingcited by 9
  7. 2022Byzantines can also Learn from History: Fall of Centered Clipping in Federated Learningcited by 8
  8. 2024PALM: Pushing Adaptive Learning Rate Mechanisms for Continual Test-Time Adaptationcited by 8
  9. 2021Visualizing the Emergence of Intermediate Visual Patterns in DNNscited by 7
  10. 2022Weight Expansion: A New Perspective on Dropout and Generalizationcited by 6
  11. 2025The Right to be Forgotten in Pruning: Unveil Machine Unlearning on Sparse Modelscited by 5
  12. 2021Optimization Variance: Exploring Generalization Properties of DNNscited by 4
  13. 2023Systematic Investigation of Sparse Perturbed Sharpness-Aware Minimization Optimizercited by 4
  14. 2021Perturbated Gradients Updating within Unit Space for Deep Learningcited by 3
  15. 2024How Do Training Methods Influence the Utilization of Vision Models?cited by 3
  16. 2024Towards Optimal Adapter Placement for Efficient Transfer Learningcited by 3
  17. 2022Leveraging Unlabeled Data to Track Memorizationcited by 2
  18. 2024A Margin-based Multiclass Generalization Bound via Geometric Complexitycited by 2
  19. 2024Sharpness-Aware Black-Box Optimizationcited by 2
  20. 2021Exploring Heterogeneous Characteristics of Layers in ASR Models for More Efficient Trainingcited by 1
  21. 2024In-Distribution Consistency Regularization Improves the Generalization of Quantization-Aware Trainingcited by 1
  22. 2024AdaRank: Disagreement Based Module Rank Prediction for Low-rank Adaptationcited by 1
  23. 2026Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directionscited by 1
  24. 2024Majority Kernels: An Approach to Leverage Big Model Dynamics for Efficient Small Model Trainingcited by 0
  25. 2024In Search of the Successful Interpolation: On the Role of Sharpness in CLIP Generalizationcited by 0
  26. 2025Seeking Flat Minima over Diverse Surrogates for Improved Adversarial Transferability: A Theoretical Framework and Algorithmic Instantiationcited by 0
  27. 2025Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learningcited by 0
  28. 2026Incorruptible Neural Networks: Training Models that can Generalize to Large Internal Perturbationscited by 0