# The Ramifications of Making Deep Neural Networks Compact

arXiv: https://arxiv.org/abs/2006.15098 · HTML: https://mlsanity.com/paper/2006.15098/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jun 2020.

## Built on by (most cited first, 6 of 6)

- [Machine Learning for Microcontroller-Class Hardware: A Review](https://mlsanity.com/paper/2205.14550/) (2022)
- [Enabling Resource-efficient AIoT System with Cross-level Optimization: A survey](https://mlsanity.com/paper/2309.15467/) (2023)
- [Deep neural networks with controlled variable selection for the identification of putative causal genetic variants](https://mlsanity.com/paper/2109.14719/) (2021)
- [Estimating Deep Learning energy consumption based on model architecture and training environment](https://mlsanity.com/paper/2307.05520/) (2023)
- [AdaSpring: Context-adaptive and Runtime-evolutionary Deep Model Compression for Mobile Applications](https://mlsanity.com/paper/2101.11800/) (2021)
- [6thGrid-Net: Unified Remote Sensing Image Dehazing Based on Color Restoration and Edge-Preserving](https://mlsanity.com/paper/2604.24149/) (2026)

## Abstract (arXiv)

The recent trend in deep neural networks (DNNs) research is to make the networks more compact. The motivation behind designing compact DNNs is to improve energy efficiency since by virtue of having lower memory footprint, compact DNNs have lower number of off-chip accesses which improves energy efficiency. However, we show that making DNNs compact has indirect and subtle implications which are not well-understood. Reducing the number of parameters in DNNs increases the number of activations which, in turn, increases the memory footprint. We evaluate several recently-proposed compact DNNs on Tesla P100 GPU and show that their "activations to parameters ratio" ranges between 1.4 to 32.8. Further, the "memory-footprint to model size ratio" ranges between 15 to 443. This shows that a higher number of activations causes large memory footprint which increases on-chip/off-chip data movements. Furthermore, these parameter-reducing techniques reduce the arithmetic intensity which increases on-chip/off-chip memory bandwidth requirement. Due to these factors, the energy efficiency of compact DNNs may be significantly reduced which is against the original motivation for designing compact DNNs.