mlsanityData to 7 Oct 2026

Vanishing Contributions: A Unified Framework for Smooth and Iterative Model Compression

arXiv 2510.09696 ↗First posted 9 Oct 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The increasing scale of Deep Neural Networks (DNNs) introduces the need for compression techniques such as pruning, quantization, and low-rank decomposition. While these methods are very effective at reducing memory, computation, and energy consumption, they may introduce severe accuracy degradation, which is often mitigated by using iterative, gradual compression. However, different compression techniques require distinct iterative approaches, and some result in unstable, discontinuous model fine-tuning. We introduce Vanishing Contributions (VCON), a unified framework for the smooth, iterative transition of DNNs into a compressed form. Rather than replacing the original network directly with its compressed version, VCON executes both in parallel during fine-tuning. The contribution of the original (uncompressed) model is progressively reduced, while that of the compressed model is gradually increased. This affine combination allows the network to slowly adapt, improving stability and mitigating accuracy degradation. We evaluate VCON on computer vision and natural language processing benchmarks, using multiple compression strategies. In most settings, our framework improves accuracy over post-shot and iterative baselines. Typical gains exceed 1%, while some configuration exhibits improvements above 15%. VCON is thus compatible with existing compression techniques and consistently improves performance across diverse tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 54 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

42 of 54 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  7. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  9. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  10. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  11. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  12. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
30 more
  1. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
  2. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  3. 2015Tensorizing Neural Networkscited by 246
  4. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  5. 2013Predicting Parameters in Deep Learningcited by 223
  6. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 211
  7. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  8. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  9. 2021Group Fisher Pruning for Practical Network Compressioncited by 130
  10. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  11. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  12. 2023Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networkscited by 80
  13. 2018A Survey on Methods and Theories of Quantized Neural Networkscited by 75
  14. 2020Methods for Pruning Deep Neural Networkscited by 49
  15. 2021Manifold Regularized Dynamic Network Pruningcited by 43
  16. 2022AdaBin: Improving Binary Neural Networks with Adaptive Binary Setscited by 38
  17. 2020A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutionscited by 32
  18. 2022DropNet: Reducing Neural Network Complexity via Iterative Pruningcited by 29
  19. 2022OPQ: Compressing Deep Neural Networks with One-shot Pruning-Quantizationcited by 19
  20. 2024TernaryLLM: Ternarized Large Language Modelcited by 15
  21. 2021Neural Network Quantization for Efficient Inference: A Surveycited by 13
  22. 2024SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compressioncited by 12
  23. 2020Embedding Compression with Isotropic Iterative Quantizationcited by 10
  24. 2022TerViT: An Efficient Ternary Vision Transformercited by 10
  25. 2024BiPer: Binary Neural Networks using a Periodic Functioncited by 9
  26. 2024S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-trainingcited by 7
  27. 2023Progressive Channel-Shrinking Networkcited by 4
  28. 2019Progressive DNN Compression: A Key to Achieve Ultra-High Weight Pruning and Quantization Rates using ADMMcited by 3
  29. 2018Iterative Low-Rank Approximation for CNN Compressioncited by 2
  30. 2023Towards Higher Ranks via Adversarial Weight Pruningcited by 2

Built on by

Not cited yet.