# Analysis of memory consumption by neural networks based on hyperparameters

arXiv: https://arxiv.org/abs/2110.11424 · HTML: https://mlsanity.com/paper/2110.11424/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 19 references that are in the map.
- First posted on arXiv 21 Oct 2021.

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Point-Voxel CNN for Efficient 3D Deep Learning](https://mlsanity.com/paper/1907.03739/) (2019)
- [Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications](https://mlsanity.com/paper/1511.06530/) (2015)
- [Bayesian Compression for Deep Learning](https://mlsanity.com/paper/1705.08665/) (2017)
- [Chainer: A Deep Learning Framework for Accelerating the Research Cycle](https://mlsanity.com/paper/1908.00213/) (2019)
- [Memory-Efficient Implementation of DenseNets](https://mlsanity.com/paper/1707.06990/) (2017)
- [Efficient Memory Management for GPU-based Deep Learning Systems](https://mlsanity.com/paper/1903.06631/) (2019)

## Built on by (most cited first, 1 of 1)

- [Towards a multi-stakeholder value-based assessment framework for algorithmic systems](https://mlsanity.com/paper/2205.04525/) (2022)

## Abstract (arXiv)

Deep learning models are trained and deployed in multiple domains. Increasing usage of deep learning models alarms the usage of memory consumed while computation by deep learning models. Existing approaches for reducing memory consumption like model compression, hardware changes are specific. We propose a generic analysis of memory consumption while training deep learning models in comparison with hyperparameters used for training. Hyperparameters which includes the learning rate, batchsize, number of hidden layers and depth of layers decide the model performance, accuracy of the model. We assume the optimizers and type of hidden layers as a known values. The change in hyperparamaters and the number of hidden layers are the variables considered in this proposed approach. For better understanding of the computation cost, this proposed analysis studies the change in memory consumption with respect to hyperparameters as main focus. This results in general analysis of memory consumption changes during training when set of hyperparameters are altered.