# Is Oracle Pruning the True Oracle?

arXiv: https://arxiv.org/abs/2412.00143 · HTML: https://mlsanity.com/paper/2412.00143/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 88 references that are in the map.
- First posted on arXiv 28 Nov 2024.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1808.06866/) (2018)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Comparing Rewinding and Fine-tuning in Neural Network Pruning](https://mlsanity.com/paper/2003.02389/) (2020)

## Built on by (most cited first, 7 of 7)

- [Efficient Reasoning Models: A Survey](https://mlsanity.com/paper/2504.10903/) (2025)
- [ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Maps](https://mlsanity.com/paper/2505.18675/) (2025)
- [RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning](https://mlsanity.com/paper/2510.02240/) (2025)
- [Autoregressive Image Generation with Randomized Parallel Decoding](https://mlsanity.com/paper/2503.10568/) (2025)
- [OBS-Diff: Accurate Pruning For Diffusion Models in One-Shot](https://mlsanity.com/paper/2510.06751/) (2025)
- [Which Heads Matter for Reasoning? RL-Guided KV Cache Compression](https://mlsanity.com/paper/2510.08525/) (2025)
- [Cross-Resolution Diffusion Models via Network Pruning](https://mlsanity.com/paper/2604.05524/) (2026)

## Abstract (arXiv)

Oracle pruning, which selects unimportant weights by minimizing the pruned train loss, has served as the foundation for most neural network pruning methods for over thirty-five years, while few (if any) have thought about how much the foundation really holds. This paper, for the first time, attempts to systematically examine its validity on deep neural networks through empirical correlation analyses and provides meta-framework reflections on the field of neural network pruning. Specifically, this paper focuses on the pruning algorithms with three stages: training, pruning, and retraining. We analyze the correlation in model performance before and after the retraining stage. Extensive experiments (37K models are trained) across a wide spectrum of models (LeNet5, VGG, ResNets, ViT, MLLM) and datasets (MNIST, CIFAR10/CIFAR100, ImageNet-1K, MLLM data) are conducted. For large-scale experiments, we adopt approximate oracle pruning due to the prohibitive cost of exact oracle pruning. The results point to a counterintuitive conclusion: for deep learning models of nontrivial size (already at the scale of ResNet56 on CIFAR-10), pre-retraining performance is negligibly correlated with post-retraining performance. In other words, the weights identified by oracle pruning can scarcely guarantee strong performance following retraining. This further suggests that existing works that derive pruning criteria from oracle pruning may rest on a questionable foundational premise. Further studies suggest that rising task complexity is a primary factor behind the invalidity of oracle pruning nowadays. Finally, given the evidence, we argue that the retraining stage in a pruning algorithm should be accounted for when developing pruning criteria.