mlsanityData to 14 Oct 2026

SAFE: Finding Sparse and Flat Minima to Improve Pruning

arXiv 2506.06866 ↗First posted 7 Jun 2025cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Sparsifying neural networks often suffers from seemingly inevitable performance degradation, and it remains challenging to restore the original performance despite much recent progress. Motivated by recent studies in robust optimization, we aim to tackle this problem by finding subnetworks that are both sparse and flat at the same time. Specifically, we formulate pruning as a sparsity-constrained optimization problem where flatness is encouraged as an objective. We solve it explicitly via an augmented Lagrange dual approach and extend it further by proposing a generalized projection operation, resulting in novel pruning methods called SAFE and its extension, SAFE^+. Extensive evaluations on standard image classification and language modeling tasks reveal that SAFE consistently yields sparse networks with improved generalization performance, which compares competitively to well-established baselines. In addition, SAFE demonstrates resilience to noisy data, making it well-suited for real-world conditions.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 60 of its 74 references that are in the map. First posted on arXiv 7 Jun 2025.

Builds on

60 of 74 references.

References · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  11. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  12. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
38 more
  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  2. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  5. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  6. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  7. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  8. 2017Exploring Generalization in Deep Learningcited by 600
  9. 2018Rethinking the Value of Network Pruningcited by 599
  10. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
  11. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  12. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  13. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  14. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  15. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  16. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  17. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  18. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  19. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  20. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 237
  21. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  22. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 151
  23. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
  24. 2020Dynamic Model Pruning with Feedbackcited by 125
  25. 2018A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multiplierscited by 120
  26. 2020Optimal Regularization Can Mitigate Double Descentcited by 109
  27. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  28. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  29. 2015Global Convergence of ADMM in Nonconvex Nonsmooth Optimizationcited by 98
  30. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  31. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 59
  32. 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
  33. 2022Anticorrelated Noise Injection for Improved Generalizationcited by 42
  34. 2023Fast as CHITA: Neural Network Pruning with Combinatorial Optimizationcited by 40
  35. 2022Sparse Double Descent: Where Network Pruning Aggravates Overfittingcited by 26
  36. 2024On the Duality Between Sharpness-Aware Minimization and Adversarial Trainingcited by 24
  37. 2024A Survey of Lottery Ticket Hypothesiscited by 24
  38. 2024Why is SAM Robust to Label Noise?cited by 21

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMscited by 9
  2. 2025The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMMcited by 5
  3. 2025AL-CoLe: Augmented Lagrangian for Constrained Learningcited by 2
  4. 2025S2AP: Score-space Sharpness Minimization for Adversarial Pruningcited by 0
  5. 2026HORST: Composing Optimizer Geometries for Sparse Transformer Trainingcited by 0