A Metric Driven Approach to Mixed Precision Training
Not on the map. Why?
Abstract (arXiv)
As deep learning methodologies have developed, it has been generally agreed that increasing neural network size improves model quality. However, this is at the expense of memory and compute requirements, which also need to be increased. Various efficiency techniques have been proposed to rein in hardware costs, one being the use of low precision numerics. Recent accelerators have introduced several different 8-bit data types to help accommodate DNNs in terms of numerics. In this paper, we identify a metric driven methodology to aid in the choice of numerics. We demonstrate how such a methodology can help scale training of a language representation model. The technique can be generalized to other model architectures.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 16 references that are in the map. First posted on arXiv 6 Aug 2024.
Builds on
11 of 16 references.
References · 11
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
- 2017Mixed Precision Trainingcited by 1,156
- 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
- 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
- 2022FP8 Formats for Deep Learningcited by 196
- 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
- 2016Pruning Convolutional Neural Networks for Resource Efficient Inferencecited by 48
- 2021Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizercited by 2
Built on by
Not cited yet.