mlsanityData to 14 Oct 2026

The Quest of Finding the Antidote to Sparse Double Descent

arXiv 2308.16596 ↗First posted 31 Aug 2023cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In energy-efficient schemes, finding the optimal size of deep learning models is very important and has a broad impact. Meanwhile, recent studies have reported an unexpected phenomenon, the sparse double descent: as the model's sparsity increases, the performance first worsens, then improves, and finally deteriorates. Such a non-monotonic behavior raises serious questions about the optimal model's size to maintain high performance: the model needs to be sufficiently over-parametrized, but having too many parameters wastes training resources. In this paper, we aim to find the best trade-off efficiently. More precisely, we tackle the occurrence of the sparse double descent and present some solutions to avoid it. Firstly, we show that a simple ℓ2 regularization method can help to mitigate this phenomenon but sacrifices the performance/sparsity compromise. To overcome this problem, we then introduce a learning scheme in which distilling knowledge regularizes the student model. Supported by experimental results achieved using typical image classification setups, we show that this approach leads to the avoidance of such a phenomenon.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 47 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

33 of 47 references.

References · 33

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  6. 2016Sequence-Level Knowledge Distillationcited by 801
  7. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  8. 2019The State of Sparsity in Deep Neural Networkscited by 476
  9. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 256
  10. 2018Dimensionality-Driven Learning with Noisy Labelscited by 173
  11. 2020Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Modelscited by 169
  12. 2017Learning from Noisy Labels with Distillationcited by 161
21 more
  1. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 134
  2. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 130
  3. 2019Harmless interpolation of noisy data in regressioncited by 121
  4. 2020Optimal Regularization Can Mitigate Double Descentcited by 109
  5. 2018A jamming transition from under- to over-parametrization affects loss landscape and generalizationcited by 67
  6. 2018The jamming transition as a paradigm to understand the loss landscape of deep neural networkscited by 62
  7. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  8. 2020Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networkscited by 40
  9. 2016Deep Model Compression: Distilling Knowledge from Noisy Teacherscited by 36
  10. 2022Sparse Double Descent: Where Network Pruning Aggravates Overfittingcited by 26
  11. 2020Solvable Model for Inheriting the Regularization through Knowledge Distillationcited by 16
  12. 2022To update or not to update? Neurons at equilibrium in deep modelscited by 13
  13. 2021Prune Your Model Before Distill Itcited by 12
  14. 2021Frugal Machine Learningcited by 11
  15. 2022Multiple Descent in the Multiple Random Feature Modelcited by 10
  16. 2022Regularization-wise double descent: Why it occurs and how to eliminate itcited by 9
  17. 2022Blind Knowledge Distillation for Robust Image Classificationcited by 9
  18. 2021Distilling Double Descentcited by 7
  19. 2023DSD2: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free?cited by 6
  20. 2023Can we avoid Double Descent in Deep Neural Networks?cited by 2
  21. 2023Sparse Double Descent in Vision Transformers: real or phantom threat?cited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Accurate Neural Network Pruning Requires Rethinking Sparse Optimization
  2. 2023Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle
  3. 2024Effective Interplay between Sparsity and Quantization: From Theory to Practice
  4. 2025Principled Approximation Methods for Efficient and Scalable Deep Learning
  5. 2021Multi-scale Feature Learning Dynamics: Insights for Double Descent
  6. 2021SparseDNN: Fast Sparse Deep Learning Inference on CPUs
  7. 2021Compact and Optimal Deep Learning with Recurrent Parameter Generators
  8. 2022Training Thinner and Deeper Neural Networks: Jumpstart Regularization