mlsanityData to 14 Oct 2026

Decoupled Weight Decay for Any p Norm

arXiv 2404.10824 ↗First posted 16 Apr 2024cs.LG cs.AI cs.NEcited by 4

Not on the map. Why?

Abstract (arXiv)

With the success of deep neural networks (NNs) in a variety of domains, the computational and storage requirements for training and deploying large NNs have become a bottleneck for further improvements. Sparsification has consequently emerged as a leading approach to tackle these issues. In this work, we consider a simple yet effective approach to sparsification, based on the Bridge, or Lp regularization during training. We introduce a novel weight decay scheme, which generalizes the standard L2 weight decay to any p norm. We show that this scheme is compatible with adaptive optimizers, and avoids the gradient divergence associated with 0<p<1 norms. We empirically demonstrate that it leads to highly sparse networks, while maintaining generalization performance comparable to standard L2 regularization.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 21 of its 56 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

21 of 56 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  7. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,512
  8. 2020Variational Quantum Algorithmscited by 692
  9. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  10. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  11. 2021Accelerating Sparse Deep Neural Networkscited by 223
  12. 2018Efficient Neural Audio Synthesiscited by 219
9 more
  1. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  2. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  3. 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 110
  4. 2017Soft Weight-Sharing for Neural Network Compressioncited by 99
  5. 2018Faster gaze prediction with dense networks and Fisher pruningcited by 94
  6. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  7. 2018Bridgeout: stochastic bridge regularization for deep neural networkscited by 4
  8. 2022Deterministic Bridge Regression for Compressive Classificationcited by 1
  9. 2023Lp Quasi-norm Minimization: Algorithm and Applicationscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetriescited by 14
  2. 2024Fine-Tuning and Evaluating Open-Source Large Language Models for the Army Domaincited by 6
  3. 2025Global Minimizers of ℓp-Regularized Objectives Yield the Sparsest ReLU Neural Networkscited by 0
  4. 2026PQuantML: A Tool for End-to-End Hardware-aware Model Compressioncited by 0