How Low Can We Go: Trading Memory for Error in Low-Precision Training
Not on the map. Why?
Abstract (arXiv)
Low-precision arithmetic trains deep learning models using less energy, less memory and less time. However, we pay a price for the savings: lower precision may yield larger round-off error and hence larger prediction error. As applications proliferate, users must choose which precision to use to train a new model, and chip manufacturers must decide which precisions to manufacture. We view these precision choices as a hyperparameter tuning problem, and borrow ideas from meta-learning to learn the tradeoff between memory and error. In this paper, we introduce Pareto Estimation to Pick the Perfect Precision (PEPPP). We use matrix factorization to find non-dominated configurations (the Pareto frontier) with a limited number of network evaluations. For any given memory budget, the precision that minimizes error is a point on this frontier. Practitioners can use the frontier to trade memory for error and choose the best precision for their goals.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 50 references that are in the map. First posted on arXiv 17 Jun 2021.
Builds on
25 of 50 references.
References · 25
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2021Convex optimizationcited by 2,479
- 2017Mixed Precision Trainingcited by 1,135
- 2020Meta-Learning in Neural Networks: A Surveycited by 944
- 2018A Tutorial on Bayesian Optimizationcited by 734
- 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
- 2015Deep Learning with Limited Numerical Precisioncited by 368
- 2018Meta-Learning: A Surveycited by 254
- 2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 226
- 2020The Hardware Lotterycited by 160
- 2018Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Searchcited by 122
- 2014Matrix Completion and Low-Rank SVD via Fast Alternating Least Squarescited by 104
13 more
- 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 100
- 2020Rethinking Differentiable Search for Mixed-Precision Neural Networkscited by 79
- 2019Low-Memory Neural Network Training: A Technical Reportcited by 56
- 2017Probabilistic Matrix Factorization for Automated Machine Learningcited by 31
- 2018OBOE: Collaborative Filtering for AutoML Model Selectioncited by 31
- 2019Missing Not at Random in Matrix Completion: The Effectiveness of Estimating Missingness Probabilities Under a Low Nuclear Norm Assumptioncited by 29
- 2019QPyTorch: A Low-Precision Arithmetic Simulation Frameworkcited by 21
- 2018High-Accuracy Low-Precision Trainingcited by 20
- 2020Extending and Analyzing Self-Supervised Learning Across Domainscited by 20
- 2016Understanding the Impact of Precision Quantization on the Accuracy and Energy of Neural Networkscited by 17
- 2021A Bregman Learning Framework for Sparse Neural Networkscited by 15
- 2019Cheetah: Mixed Low-Precision Hardware & Software Co-Design Framework for DNNs on the Edgecited by 8
- 2016The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learningcited by 2
Built on by
Citing papers per year: 2023: 1; 2024: 0; 2025: 0; 2026: 1.
2 papers.