mlsanityData to 13 Oct 2026

Compression-aware Training of Neural Networks using Frank-Wolfe

arXiv 2205.11921 ↗First posted 24 May 2022cs.LG math.OCcited by 16

Not on the map. Why?

Abstract (arXiv)

Many existing Neural Network pruning approaches rely on either retraining or inducing a strong bias in order to converge to a sparse solution throughout training. A third paradigm, 'compression-aware' training, aims to obtain state-of-the-art dense models that are robust to a wide range of compression ratios using a single dense training run while also avoiding retraining. We propose a framework centered around a versatile family of norm constraints and the Stochastic Frank-Wolfe (SFW) algorithm that encourage convergence to well-performing solutions while inducing robustness towards convolutional filter pruning and low-rank matrix decomposition. Our method is able to outperform existing compression-aware approaches and, in the case of low-rank matrix decomposition, it also requires significantly less computational resources than approaches based on nuclear-norm regularization. Our findings indicate that dynamically adjusting the learning rate of SFW, as suggested by Pokutta et al. (2020), is crucial for convergence and robustness of SFW-trained models and we establish a theoretical foundation for that practice.

Key facts. Cited by 16 papers in the map (data through 13 Oct 2026). Builds on 62 of its 96 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

62 of 96 references.

References · 62

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,476
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,991
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,435
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,601
  6. 2016Wide Residual Networkscited by 3,063
  7. 2016Pyramid Scene Parsing Networkcited by 2,474
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,017
  9. 2016Understanding deep learning requires rethinking generalizationcited by 1,666
  10. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,248
  11. 2016Pruning Filters for Efficient ConvNetscited by 1,138
  12. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 783
38 more
  1. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 602
  2. 2018Rethinking the Value of Network Pruningcited by 597
  3. 2020What is the State of Neural Network Pruning?cited by 582
  4. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  5. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 525
  6. 2019Rigging the Lottery: Making All Tickets Winnerscited by 493
  7. 2019The State of Sparsity in Deep Neural Networkscited by 474
  8. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 315
  9. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 305
  10. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 240
  11. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 201
  12. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 162
  13. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 150
  14. 2020Dynamic Model Pruning with Feedbackcited by 125
  15. 2017L2 Regularization versus Batch and Weight Normalizationcited by 123
  16. 2020Layer-adaptive sparsity for the Magnitude-based Pruningcited by 123
  17. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  18. 2015Convolutional neural networks with low-rank regularizationcited by 102
  19. 2020EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruningcited by 80
  20. 2019Global Sparse Momentum SGD for Pruning Very Deep Neural Networkscited by 70
  21. 2016Learning the Number of Neurons in Deep Networkscited by 69
  22. 2020Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layerscited by 68
  23. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  24. 2016Convergence Rate of Frank-Wolfe for Non-Convex Objectivescited by 60
  25. 2019Discovering Neural Wiringscited by 56
  26. 2019The Generalization-Stability Tradeoff In Neural Network Pruningcited by 54
  27. 2020Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsificationcited by 52
  28. 2017Compression-aware Training of Deep Networkscited by 46
  29. 2016Variance-Reduced and Projection-Free Stochastic Optimizationcited by 40
  30. 2016Stochastic Frank-Wolfe Methods for Nonconvex Optimizationcited by 40
  31. 2019Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraintscited by 37
  32. 2021Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decompositioncited by 33
  33. 2020TRP: Trained Rank Pruning for Efficient Deep Neural Networkscited by 32
  34. 2018Projection-Free Online Optimization with Stochastic Gradient: From Convexity to Submodularitycited by 31
  35. 2020Growing Efficient Deep Networks by Structured Continuous Sparsificationcited by 28
  36. 2021Network Pruning That Matters: A Case Study on Retraining Variantscited by 25
  37. 2017Coordinating Filters for Faster Deep Neural Networkscited by 24
  38. 2016LazySVD: Even Faster SVD Decomposition Yet Without Agonizing Paincited by 22

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 1; 2025: 6; 2026: 3.

16 papers.

Sort:

By area:

  1. 2023Sparse Model Soups: A Recipe for Improved Pruning via Model Averagingcited by 20
  2. 2023PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMscited by 14
  3. 2022CrAM: A Compression-Aware Minimizercited by 7
  4. 2023Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Trainingcited by 5
  5. 2024Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance Reductioncited by 5
  6. 2025Approximating Latent Manifolds in Neural Networks via Vanishing Idealscited by 5
  7. 2025SparseSwaps: Tractable LLM Pruning Mask Refinement at Scalecited by 4
  8. 2026NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Trainingcited by 4
  9. 2026The Agentic Researcher: A Practical Guide to AI-Assisted Research in Mathematics and Machine Learningcited by 4
  10. 2025Implicit Bias in Matrix Factorization and its Explicit Realization in a New Architecturecited by 1
  11. 2025Don't Be Greedy, Just Relax! Pruning LLMs via Frank-Wolfecited by 1
  12. 2026SLORR: Simple and Efficient In-Training Low-Rank Regularizationcited by 1
4 more
  1. 2022Renormalized Sparse Neural Network Pruningcited by 0
  2. 2023ELSA: Partial Weight Freezing for Overhead-Free Sparse Network Deploymentcited by 0
  3. 2025Gibbs randomness-compression propositioncited by 0
  4. 2025Compression Aware Certified Trainingcited by 0