Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators
Not on the map. Why?
Abstract (arXiv)
In this paper, we propose a mixed-precision convolution unit architecture which supports different integer and floating point (FP) precisions. The proposed architecture is based on low-bit inner product units and realizes higher precision based on temporal decomposition. We illustrate how to integrate FP computations on integer-based architecture and evaluate overheads incurred by FP arithmetic support. We argue that alignment and addition overhead for FP inner product can be significant since the maximum exponent difference could be up to 58 bits, which results into a large alignment logic. To address this issue, we illustrate empirically that no more than 26-bitproduct bits are required and up to 8-bit of alignment is sufficient in most inference cases. We present novel optimizations based on the above observations to reduce the FP arithmetic hardware overheads. Our empirical results, based on simulation and hardware implementation, show significant reduction in FP16 overhead. Over typical mixed precision implementation, the proposed architecture achieves area improvements of up to 25% in TFLOPS/mm2and up to 46% in TOPS/mm2with power efficiency improvements of up to 40% in TFLOPS/Wand up to 63% in TOPS/W.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 53 of its 90 references that are in the map. First posted on arXiv 27 Jan 2021.
Builds on
53 of 90 references.
References · 53
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
- 2017Mixed Precision Trainingcited by 1,135
- 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
- 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
- 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
- 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
- 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
- 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
38 more
- 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
- 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
- 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
- 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
- 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
- 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
- 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
- 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
- 2016Trained Ternary Quantizationcited by 203
- 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 185
- 2016A guide to convolution arithmetic for deep learningcited by 182
- 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 157
- 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
- 2018Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Searchcited by 122
- 2017Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networkscited by 95
- 2018Training and Inference with Integers in Deep Neural Networkscited by 84
- 2020Post-Training Piecewise Linear Quantization for Deep Neural Networkscited by 77
- 2017Towards Effective Low-bitwidth Convolutional Neural Networkscited by 76
- 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
- 2018Value-aware Quantization for Training and Inference of Neural Networkscited by 49
- 2018Mixed Precision Training of Convolutional Neural Networks using Integer Operationscited by 45
- 2017Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networkscited by 43
- 2018Training DNNs with Hybrid Block Floating Pointcited by 39
- 2018Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentationcited by 37
- 2019Training Quantized Neural Networks with a Full-precision Auxiliary Modulecited by 35
- 2016Bit-pragmatic Deep Neural Network Computingcited by 30
- 2018Neural Cache: Bit-Serial In-Cache Acceleration of Deep Neural Networkscited by 29
- 2020Fractional Skipping: Towards Finer-Grained Dynamic CNN Inferencecited by 23
- 2020Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networkscited by 23
- 2018A Quantization-Friendly Separable Convolution for MobileNetscited by 22
- 2018Rethinking floating point for deep learningcited by 22
- 2018High-Accuracy Low-Precision Trainingcited by 20
- 2018Quantization for Rapid Deployment of Deep Neural Networkscited by 20
- 2017Loom: Exploiting Weight and Activation Precisions to Accelerate Convolutional Neural Networkscited by 11
- 2018GANAX: A Unified MIMD-SIMD Acceleration for Generative Adversarial Networkscited by 11
- 2016Dark Memory and Accelerator-Rich System Optimization in the Dark Silicon Eracited by 8
- 2016Accelerating Deep Convolutional Networks using low-precision and sparsitycited by 8
- 2019Training Deep Neural Networks Using Posit Number Systemcited by 6
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2022: 1; 2023: 2.
3 papers.