Nesting Forward Automatic Differentiation for Memory-Efficient Deep Neural Network Training
Not on the map. Why?
Abstract (arXiv)
An activation function is an element-wise mathematical function and plays a crucial role in deep neural networks (DNN). Many novel and sophisticated activation functions have been proposed to improve the DNN accuracy but also consume massive memory in the training process with back-propagation. In this study, we propose the nested forward automatic differentiation (Forward-AD), specifically for the element-wise activation function for memory-efficient DNN training. We deploy nested Forward-AD in two widely-used deep learning frameworks, TensorFlow and PyTorch, which support the static and dynamic computation graph, respectively. Our evaluation shows that nested Forward-AD reduces the memory footprint by up to 1.97x than the baseline model and outperforms the recomputation by 20% under the same memory reduction ratio.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 22 Sep 2022.
Builds on
21 of 38 references.
References · 21
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2017Mask R-CNNcited by 7,259
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
- 2015Automatic differentiation in machine learning: a surveycited by 846
- 2016Training Deep Nets with Sublinear Memory Costcited by 825
- 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
- 2019Mish: A Self Regularized Non-Monotonic Activation Functioncited by 141
- 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 90
- 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
9 more
- 2022SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximationcited by 51
- 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 42
- 2017Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networkscited by 36
- 2020Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsitycited by 30
- 2021Dual-side Sparse Tensor Corecited by 27
- 2019Adversarial Defense Through Network Profiling Based Path Extractioncited by 22
- 2018Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagatorcited by 14
- 2020Ptolemy: Architecture Support for Robust Deep Learningcited by 12
- 2019Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUscited by 10
Built on by
Citing papers per year: 2023: 1.
1 paper.