# Memory-Efficient Implementation of DenseNets

arXiv: https://arxiv.org/abs/1707.06990 · HTML: https://mlsanity.com/paper/1707.06990/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2017.

## Built on by (most cited first, 17 of 17)

- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecycle](https://mlsanity.com/paper/2303.01980/) (2023)
- [SparX: A Sparse Cross-Layer Connection Mechanism for Hierarchical Vision Mamba and Transformer Networks](https://mlsanity.com/paper/2409.09649/) (2024)
- [DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs](https://mlsanity.com/paper/2403.19588/) (2024)
- [Adversarial Attacks on Image Classification Models: FGSM and Patch Attacks and their Impact](https://mlsanity.com/paper/2307.02055/) (2023)
- [Revisiting consistency for semi-supervised semantic segmentation](https://mlsanity.com/paper/2106.07075/) (2021)
- [Data-Free Dynamic Compression of CNNs for Tractable Efficiency](https://mlsanity.com/paper/2309.17211/) (2023)
- [Bio-inspired Min-Nets Improve the Performance and Robustness of Deep Networks](https://mlsanity.com/paper/2201.02149/) (2022)
- [TorchQL: A Programming Framework for Integrity Constraints in Machine Learning](https://mlsanity.com/paper/2308.06686/) (2023)
- [ReffAKD: Resource-efficient Autoencoder-based Knowledge Distillation](https://mlsanity.com/paper/2404.09886/) (2024)
- [AFINet: Attentive Feature Integration Networks for Image Classification](https://mlsanity.com/paper/2105.04354/) (2021)
- [Analysis of memory consumption by neural networks based on hyperparameters](https://mlsanity.com/paper/2110.11424/) (2021)
- [Memory-Efficient Sequential Pattern Mining with Hybrid Tries](https://mlsanity.com/paper/2202.06834/) (2022)
- [FetusMapV2: Enhanced Fetal Pose Estimation in 3D Ultrasound](https://mlsanity.com/paper/2310.19293/) (2023)
- [All Attention U-NET for Semantic Segmentation of Intracranial Hemorrhages In Head CT Images](https://mlsanity.com/paper/2312.10483/) (2023)
- [On Unsupervised Image-to-image translation and GAN stability](https://mlsanity.com/paper/2403.09646/) (2023)
- [Evaluating Federated Learning approaches for mammography under breast density heterogeneity](https://mlsanity.com/paper/2605.09137/) (2026)

## Abstract (arXiv)

The DenseNet architecture is highly computationally efficient as a result of feature reuse. However, a naive DenseNet implementation can require a significant amount of GPU memory: If not properly managed, pre-activation batch normalization and contiguous convolution operations can produce feature maps that grow quadratically with network depth. In this technical report, we introduce strategies to reduce the memory consumption of DenseNets during training. By strategically using shared memory allocations, we reduce the memory cost for storing feature maps from quadratic to linear. Without the GPU memory bottleneck, it is now possible to train extremely deep DenseNets. Networks with 14M parameters can be trained on a single GPU, up from 4M. A 264-layer DenseNet (73M parameters), which previously would have been infeasible to train, can now be trained on a single workstation with 8 NVIDIA Tesla M40 GPUs. On the ImageNet ILSVRC classification dataset, this large DenseNet obtains a state-of-the-art single-crop top-1 error of 20.26%.