mlsanityData to 7 Oct 2026

Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compression

arXiv 2502.16638 ↗First posted 23 Feb 2025cs.LG cs.AI cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

Structured pruning and quantization are fundamental techniques used to reduce the size of deep neural networks (DNNs) and typically are applied independently. Applying these techniques jointly via co-optimization has the potential to produce smaller, high-quality models. However, existing joint schemes are not widely used because of (1) engineering difficulties (complicated multi-stage processes), (2) black-box optimization (extensive hyperparameter tuning to control the overall compression), and (3) insufficient architecture generalization. To address these limitations, we present the framework GETA, which automatically and efficiently performs joint structured pruning and quantization-aware training on any DNNs. GETA introduces three key innovations: (i) a quantization-aware dependency graph (QADG) that constructs a pruning search space for generic quantization-aware DNN, (ii) a partially projected stochastic gradient method that guarantees layerwise bit constraints are satisfied, and (iii) a new joint learning strategy that incorporates interpretable relationships between pruning and quantization. We present numerical experiments on both convolutional neural networks and transformer architectures that show that our approach achieves competitive (often superior) performance compared to existing joint pruning and quantization methods.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 52 of its 81 references that are in the map. First posted on arXiv 23 Feb 2025.

Builds on

52 of 81 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  11. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
38 more
  1. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  2. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  3. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  4. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  5. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  6. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  7. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
  8. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  9. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
  10. 2023DepGraph: Towards Any Structural Pruningcited by 202
  11. 2019Towards Optimal Structured CNN Pruning via Generative Adversarial Learningcited by 145
  12. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  13. 2018A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multiplierscited by 118
  14. 2017Bayesian Compression for Deep Learningcited by 116
  15. 2020APQ: Joint Search for Network Architecture, Pruning and Quantization Policycited by 98
  16. 2020Bayesian Bits: Unifying Quantization and Pruningcited by 93
  17. 2020EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruningcited by 80
  18. 2020Highly Efficient Salient Object Detection with 100K Parameterscited by 79
  19. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  20. 2018Compressing Neural Networks using the Variational Information Bottleneckcited by 59
  21. 2019DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measurescited by 56
  22. 2023LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuningcited by 53
  23. 2019FPGA/DNN Co-Design: An Efficient Design Methodology for IoT Intelligence on the Edgecited by 48
  24. 2020Pruning Filter in Filtercited by 46
  25. 2020Differentiable Joint Pruning and Quantization for Hardware Efficiencycited by 45
  26. 2018Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compressioncited by 40
  27. 2023LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recoverycited by 37
  28. 2023OTOV2: Automatic, Generic, User-Friendlycited by 32
  29. 2018Accelerate CNN via Recursive Bayesian Pruningcited by 27
  30. 2022CADyQ: Content-Aware Dynamic Quantization for Image Super-Resolutioncited by 26
  31. 2020Orthant Based Proximal Stochastic Gradient Method for ℓ1-Regularized Optimizationcited by 19
  32. 2022OPQ: Compressing Deep Neural Networks with One-shot Pruning-Quantizationcited by 19
  33. 2019Mixed Precision DNNs: All you need is a good parametrizationcited by 18
  34. 2024Effective Interplay between Sparsity and Quantization: From Theory to Practicecited by 16
  35. 2018A Unified Framework of DNN Weight Pruning and Weight Clustering/Quantization Using ADMMcited by 10
  36. 2023Hardware-Aware DNN Compression via Diverse Pruning and Mixed-Precision Quantizationcited by 9
  37. 2016A Reduced-Space Algorithm for Minimizing ℓ1-Regularized Convex Functionscited by 7
  38. 2021Single-path Bit Sharing for Automatic Loss-aware Model Compressioncited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 11.

14 papers.

Sort:

By area:

  1. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  2. 2025WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inferencecited by 2
  3. 2025CoDeQ: End-to-End Joint Model Compression with Dead-Zone Quantizer for High-Sparsity and Low-Precision Networkscited by 2
  4. 2026StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidthscited by 2
  5. 2026Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compressioncited by 2
  6. 2026Towards Joint Quantization and Token Pruning of Vision-Language Modelscited by 1
  7. 2026Lightweight Edge Learning via Dataset Pruningcited by 0
  8. 2026pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Trainingcited by 0
  9. 2026GETA-3DGS: Automatic Joint Structured Pruning and Quantization for 3D Gaussian Splattingcited by 0
  10. 2026AI-IoT-Robotics Integration: Survey of Frameworks, Emerging Trends, and the Path Toward Connected Roboticscited by 0
  11. 2026Toward Multi-Domain and Long-Tailed Quantization via Feature Alignment and Scalingcited by 0
  12. 2026APQF: Agentic Profiling-Guided Structured Pruning and Mixed-Precision Quantization with Adaptive Fine-Tuningcited by 0
2 more
  1. 2026Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMscited by 0
  2. 2026H3DNAS: Hardware-Aware ONNX-Native 3D Point Cloud Model Compressioncited by 0