# Going Beyond Classification Accuracy Metrics in Model Compression

arXiv: https://arxiv.org/abs/2012.01604 · HTML: https://mlsanity.com/paper/2012.01604/

## Key facts

- Cited by 15 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 3 Dec 2020.

## Built on by (most cited first, 15 of 15)

- [Pruning has a disparate impact on model accuracy](https://mlsanity.com/paper/2205.13574/) (2022)
- [Accuracy is Not All You Need](https://mlsanity.com/paper/2407.09141/) (2024)
- [Can Model Compression Improve NLP Fairness](https://mlsanity.com/paper/2201.08542/) (2022)
- [Bias in Pruned Vision Models: In-Depth Analysis and Countermeasures](https://mlsanity.com/paper/2304.12622/) (2023)
- [Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithm](https://mlsanity.com/paper/2206.02976/) (2022)
- [DRoP: Distributionally Robust Data Pruning](https://mlsanity.com/paper/2404.05579/) (2024)
- [A Fair Loss Function for Network Pruning](https://mlsanity.com/paper/2211.10285/) (2022)
- [What is Left After Distillation? How Knowledge Transfer Impacts Fairness and Bias](https://mlsanity.com/paper/2410.08407/) (2024)
- [Disparate Impact on Group Accuracy of Linearization for Private Inference](https://mlsanity.com/paper/2402.03629/) (2024)
- [Understanding the Effect of the Long Tail on Neural Network Compression](https://mlsanity.com/paper/2306.06238/) (2023)
- [ArctyrEX : Accelerated Encrypted Execution of General-Purpose Applications](https://mlsanity.com/paper/2306.11006/) (2023)
- [Physics-Enhanced TinyML for Real-Time Detection of Ground Magnetic Anomalies](https://mlsanity.com/paper/2311.11452/) (2023)
- [What Operations can be Performed Directly on Compressed Arrays, and with What Error?](https://mlsanity.com/paper/2406.11209/) (2024)
- [Downsized and Compromised?: Assessing the Faithfulness of Model Compression](https://mlsanity.com/paper/2510.06125/) (2025)
- [Bias In, Bias Out? Finding Unbiased Subnetworks in Vanilla Models](https://mlsanity.com/paper/2603.05582/) (2026)

## Abstract (arXiv)

With the rise in edge-computing devices, there has been an increasing demand to deploy energy and resource-efficient models. A large body of research has been devoted to developing methods that can reduce the size of the model considerably without affecting the standard metrics such as top-1 accuracy. However, these pruning approaches tend to result in a significant mismatch in other metrics such as fairness across classes and explainability. To combat such misalignment, we propose a novel multi-part loss function inspired by the knowledge-distillation literature. Through extensive experiments, we demonstrate the effectiveness of our approach across different compression algorithms, architectures, tasks as well as datasets. In particular, we obtain up to 4.1× reduction in the number of prediction mismatches between the compressed and reference models, and up to 5.7× in cases where the reference model makes the correct prediction; all while making no changes to the compression algorithm, and minor modifications to the loss function. Furthermore, we demonstrate how inducing simple alignment between the predictions of the models naturally improves the alignment on other metrics including fairness and attributions. Our framework can thus serve as a simple plug-and-play component for compression algorithms in the future.