# Studying the impact of magnitude pruning on contrastive learning methods

arXiv: https://arxiv.org/abs/2207.00200 · HTML: https://mlsanity.com/paper/2207.00200/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 1 Jul 2022.

## Builds on (most cited first, 12 of 12)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [What Do Compressed Deep Neural Networks Forget?](https://mlsanity.com/paper/1911.05248/) (2019)
- [Deep Learning Through the Lens of Example Difficulty](https://mlsanity.com/paper/2106.09647/) (2021)
- [Understanding the effect of sparsity on neural networks robustness](https://mlsanity.com/paper/2206.10915/) (2022)
- [Class-dependent Compression of Deep Neural Networks](https://mlsanity.com/paper/1909.10364/) (2019)
- [Streamlining Tensor and Network Pruning in PyTorch](https://mlsanity.com/paper/2004.13770/) (2020)

## Built on by (most cited first, 7 of 7)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Relating tSNE and UMAP to Classical Dimensionality Reduction](https://mlsanity.com/paper/2306.11898/) (2023)
- [3D Point Cloud Network Pruning: When Some Weights Do not Matter](https://mlsanity.com/paper/2408.14601/) (2024)
- [REDS: Resource-Efficient Deep Subnetworks for Dynamic Resource Constraints](https://mlsanity.com/paper/2311.13349/) (2023)
- [Unveiling Invariances via Neural Network Pruning](https://mlsanity.com/paper/2309.08171/) (2023)
- [On the Interaction Between Model Compression and Test-Time Adaptation](https://mlsanity.com/paper/2609.03604/) (2026)
- [BaCP: Backbone Contrastive Pruning for Preserving Representations in Extremely Sparse Neural Networks](https://mlsanity.com/paper/2610.02524/) (2026)

## Abstract (arXiv)

We study the impact of different pruning techniques on the representation learned by deep neural networks trained with contrastive loss functions. Our work finds that at high sparsity levels, contrastive learning results in a higher number of misclassified examples relative to models trained with traditional cross-entropy loss. To understand this pronounced difference, we use metrics such as the number of PIEs (Hooker et al., 2019), Q-Score (Kalibhat et al., 2022), and PD-Score (Baldock et al., 2021) to measure the impact of pruning on the learned representation quality. Our analysis suggests the schedule of the pruning method implementation matters. We find that the negative impact of sparsity on the quality of the learned representation is the highest when pruning is introduced early on in the training phase.