Demystifying BERT: Implications for Accelerator Design
Not on the map. Why?
Abstract (arXiv)
Transfer learning in natural language processing (NLP), as realized using models like BERT (Bi-directional Encoder Representation from Transformer), has significantly improved language representation with models that can tackle challenging language problems. Consequently, these applications are driving the requirements of future systems. Thus, we focus on BERT, one of the most popular NLP transfer learning algorithms, to identify how its algorithmic behavior can guide future accelerator design. To this end, we carefully profile BERT training and identify key algorithmic behaviors which are worthy of attention in accelerator design. We observe that while computations which manifest as matrix multiplication dominate BERT's overall runtime, as in many convolutional neural networks, memory-intensive computations also feature prominently. We characterize these computations, which have received little attention so far. Further, we also identify heterogeneity in compute-intensive BERT computations and discuss software and possible hardware mechanisms to further optimize these computations. Finally, we discuss implications of these behaviors as networks get larger and use distributed training environments, and how techniques such as micro-batching and mixed-precision training scale. Overall, our analysis identifies holistic solutions to optimize systems for BERT-like models.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 41 of its 89 references that are in the map. First posted on arXiv 14 Apr 2021.
Builds on
41 of 89 references.
References · 41
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2014How transferable are features in deep neural networks?cited by 1,216
- 2016An overview of gradient descent optimization algorithmscited by 1,021
29 more
- 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
- 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
- 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
- 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 291
- 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 230
- 2019MLPerf Inference Benchmarkcited by 195
- 2017SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networkscited by 132
- 2019MLPerf Training Benchmarkcited by 114
- 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
- 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
- 2020A3: Accelerating Attention Mechanisms in Neural Networks with Approximationcited by 97
- 2020DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inferencecited by 54
- 2016Maximizing CNN Accelerator Efficiency Through Resource Partitioningcited by 29
- 2016Fathom: Reference Workloads for Modern Deep Learning Methodscited by 20
- 2020Automatic Horizontal Fusion for GPU Kernelscited by 17
- 2020TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inferencecited by 16
- 2019Scale MLPerf-0.6 models on Google TPU-v3 Podscited by 14
- 2016Optimizing Performance of Recurrent Neural Networks on GPUscited by 12
- 2018Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Trainingcited by 10
- 2020Exploring the limits of Concurrency in ML Training on Google TPUscited by 9
- 2021SynCron: Efficient Synchronization Support for Near-Data-Processing Architecturescited by 9
- 2021GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descentcited by 9
- 2013Optimizing CUDA Code By Kernel Fusion---Application on BLAScited by 7
- 2020Accelerating Bandwidth-Bound Deep Learning Inference with Main-Memory Acceleratorscited by 7
- 2018TBD: Benchmarking and Analyzing Deep Neural Network Trainingcited by 6
- 2020FPRaker: A Processing Element For Accelerating Neural Network Trainingcited by 6
- 2020FlexSA: Flexible Systolic Array Architecture for Efficient Pruned DNN Model Trainingcited by 5
- 2020SeqPoint: Identifying Representative Iterations of Sequence-based Neural Networkscited by 2
- 2019Demystifying the MLPerf Benchmark Suitecited by 1
Built on by
Citing papers per year: 2022: 1; 2023: 2.
3 papers.