mlsanityData to 14 Oct 2026

Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compression

arXiv 2603.18426 ↗First posted 19 Mar 2026cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

What happens when multiple compression methods are combined-does the order in which they are applied matter? Joint model compression has emerged as a powerful strategy to achieve higher efficiency by combining multiple methods such as pruning and quantization. A central but underexplored factor in joint model compression is the compression order, or the sequence of different methods within the compression pipeline. Most prior studies have either sidestepped the issue by assuming orthogonality between techniques, while a few have examined them only in highly constrained cases. Consequently, the broader role of compression order in shaping model performance remains poorly understood. In this paper, we address the overlooked problem of compression order and provide both theoretical and empirical analysis. We formulate the problem of optimizing the compression order and introduce the Progressive Intensity Hypothesis, which states that weaker perturbations should precede stronger ones. We provide theoretical guarantees showing that the relative benefit of one order increases with the underlying performance gap. Extensive experiments on both language and vision models validate the hypothesis, and further show its generality to broader setups such as multi-stage compression and mixed-precision quantization.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 60 of its 80 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

60 of 80 references.

References · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2023Mistral 7Bcited by 3,164
  11. 2016Pointer Sentinel Mixture Modelscited by 3,096
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
38 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  6. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  7. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  9. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  10. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  11. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  12. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  13. 2015Deep Learning with Limited Numerical Precisioncited by 376
  14. 2023A Survey on Model Compression for Large Language Modelscited by 269
  15. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
  16. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 261
  17. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  18. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  19. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  20. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 137
  21. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  22. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
  23. 2020APQ: Joint Search for Network Architecture, Pruning and Quantization Policycited by 99
  24. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 97
  25. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 79
  26. 2021Language Model Evaluation Beyond Perplexitycited by 78
  27. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 69
  28. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  29. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  30. 2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 35
  31. 2021Ps and Qs: Quantization-aware pruning for efficient low latency neural network inferencecited by 31
  32. 2023Gradient-Free Structured Pruning with Unlabeled Datacited by 26
  33. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23
  34. 2021PQK: Model Compression via Pruning, Quantization, and Knowledge Distillationcited by 21
  35. 2023Boost Vision Transformer with GPU-Friendly Sparsity and Quantizationcited by 20
  36. 2022OPQ: Compressing Deep Neural Networks with One-shot Pruning-Quantizationcited by 19
  37. 2024Effective Interplay between Sparsity and Quantization: From Theory to Practicecited by 19
  38. 2024LPViT: Low-Power Semi-structured Pruning for Vision Transformerscited by 16

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Towards Joint Quantization and Token Pruning of Vision-Language Modelscited by 1
  2. 2026A Closed-Loop Evaluation of Capability Loss and Recovery in Compressed Driving Policiescited by 0
  3. 2026Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Compression for Better: A General and Stable Lossless Compression Framework
  2. 2023The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models
  3. 2025A Systematic Study of Compression Ordering for Large Language Models
  4. 2025Semantic Retention and Extreme Compression in LLMs: Can We Have Both?
  5. 2026Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression
  6. 2022Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks
  7. 2026Break Through the Compression Bottleneck: From Theory to Practice
  8. 2023Towards Optimal Compression: Joint Pruning and Quantization