Data to 7 Oct 2026

A Study of BFLOAT16 for Deep Learning Training

arXiv 1905.12322 ↗First posted 29 May 2019cs.LG stat.MLcited by 227

Not on the map. Why?

Abstract (arXiv)

This paper presents the first comprehensive empirical study demonstrating the efficacy of the Brain Floating Point (BFLOAT16) half-precision format for Deep Learning training across image classification, speech recognition, language modeling, generative networks and industrial recommendation systems. BFLOAT16 is attractive for Deep Learning training for two reasons: the range of values it can represent is the same as that of IEEE 754 floating-point format (FP32) and conversion to/from FP32 is simple. Maintaining the same range as FP32 is important to ensure that no hyper-parameter tuning is required for convergence; e.g., IEEE 754 compliant half-precision floating point (FP16) requires hyper-parameter tuning. In this paper, we discuss the flow of tensors and various key operations in mixed precision training, and delve into details of operations, such as the rounding modes for converting FP32 tensors to BFLOAT16. We have implemented a method to emulate BFLOAT16 operations in Tensorflow, Caffe2, IntelCaffe, and Neon for our experiments. Our results show that deep learning training using BFLOAT16 tensors achieves the same state-of-the-art (SOTA) results across domains as FP32 tensors in the same number of iterations and with no changes to hyper-parameters.

Key facts. Cited by 227 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 20; 2023: 33; 2024: 54; 2025: 58; 2026: 48.

227 papers.

Sort:

By area:

2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
2024DeepSeek-V3 Technical Reportcited by 724
2023Large Language Models in Finance: A Surveycited by 223
2024NVILA: Efficient Frontier Visual Language Modelscited by 206
2022FP8 Formats for Deep Learningcited by 187
2023Efficient Large Language Models: A Surveycited by 183
2022Multi-lingual Evaluation of Code Generation Modelscited by 158
2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91

38 more

2024LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generationcited by 91
2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
2022Efficient Methods for Natural Language Processing: A Surveycited by 78
2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
2025Defeating the Training-Inference Mismatch via FP16cited by 62
2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 61
2023BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmarkcited by 60
2024Vision-LSTM: xLSTM as Generic Vision Backbonecited by 51
2023Intriguing Properties of Quantization at Scalecited by 48
2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 47
2024Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Modelscited by 45
2021A TinyML Platform for On-Device Continual Learning with Quantized Latent Replayscited by 37
2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
2023TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormercited by 34
2025Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreakingcited by 34
2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
2024Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attentioncited by 31
2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 29
2024Fine Tuning LLM for Enterprise: Practical Guidelines and Recommendationscited by 28
2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QAcited by 28
2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
2024Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modelingcited by 25
2024Pre-training Distillation for Large Language Models: A Design Space Explorationcited by 25
20228-bit Numerical Formats for Deep Neural Networkscited by 23
202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 23
2023Training and inference of large language models using 8-bit floating pointcited by 20
2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scalecited by 20
2025Oscillation-Reduced MXFP4 Training for Vision Transformerscited by 20
2024ATM: Adversarial Tuning Multi-agent System Makes a Robust Retrieval-Augmented Generatorcited by 19
2024BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversitycited by 19
2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
2025Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Modelscited by 18
2023Unit Scaling: Out-of-the-Box Low-Precision Trainingcited by 17
2024BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillationcited by 16
2025ChemMLLM: Chemical Multimodal Large Language Modelcited by 16
2025The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authorscited by 16
2021Accelerating SLIDE Deep Learning on Modern CPUs: Vectorization, Quantizations, Memory Optimizations, and Morecited by 15

50 most cited of 227 All 227 (JSON)