mlsanityData to 7 Oct 2026

CoNNect: Connectivity-Based Regularization for Structural Pruning

arXiv 2502.00744 ↗First posted 2 Feb 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Pruning encompasses a range of techniques aimed at increasing the sparsity of neural networks (NNs). These techniques can generally be framed as minimizing a loss function subject to an L0 norm constraint. This paper introduces CoNNect, a novel differentiable regularizer for sparse NN training that ensures connectivity between input and output layers. We prove that CoNNect approximates L0 regularization, guaranteeing maximally connected network structures while avoiding issues like layer collapse. Moreover, CoNNect is easily integrated with established structural pruning strategies. Numerical experiments demonstrate that CoNNect can improve classical pruning strategies and enhance state-of-the-art one-shot pruners, such as DepGraph and LLM-pruner.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 63 references that are in the map. First posted on arXiv 2 Feb 2025.

Builds on

42 of 63 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
30 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  2. 2017Decoupled Weight Decay Regularizationcited by 1,722
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  4. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  5. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  6. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  7. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  8. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  10. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  11. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  12. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  13. 2019The State of Sparsity in Deep Neural Networkscited by 471
  14. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  15. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  16. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
  17. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  18. 2023DepGraph: Towards Any Structural Pruningcited by 202
  19. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  20. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  21. 2015Structured Pruning of Deep Convolutional Neural Networkscited by 148
  22. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  23. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  24. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  25. 2019Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structurecited by 76
  26. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 49
  27. 2021Neural Network Training Using ℓ1-Regularization and Bi-fidelity Datacited by 16
  28. 2022spred: Solving L1 Penalty with SGDcited by 16
  29. 2021Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsitycited by 14
  30. 2022Sauron U-Net: Simple automated redundancy elimination in medical image segmentation via filter pruningcited by 5

Built on by

Not cited yet.