mlsanityData to 7 Oct 2026

Mixed Precision Training

arXiv 1710.03740 ↗First posted 10 Oct 2017cs.AI cs.LG stat.MLcited by 1,135

Not on the map. Why?

Cited by 41 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep neural networks have enabled progress in a wide variety of applications. Growing the size of the neural network typically results in improved accuracy. As model sizes grow, the memory and compute requirements for training these models also increases. We introduce a technique to train deep neural networks using half precision floating point numbers. In our technique, weights, activations and gradients are stored in IEEE half-precision format. Half-precision floating numbers have limited numerical range compared to single-precision numbers. We propose two techniques to handle this loss of information. Firstly, we recommend maintaining a single-precision copy of the weights that accumulates the gradients after each optimizer step. This single-precision copy is rounded to half-precision format during training. Secondly, we propose scaling the loss appropriately to handle the loss of information with half-precision gradients. We demonstrate that this approach works for a wide variety of models including convolution neural networks, recurrent neural networks and generative adversarial networks. This technique works for large scale models with more than 100 million parameters trained on large datasets. Using this approach, we can reduce the memory consumption of deep learning models by nearly 2x. In future processors, we can also expect a significant computation speedup using half-precision hardware units.

Key facts. Cited by 1,135 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 161; 2022: 153; 2023: 186; 2024: 226; 2025: 225; 2026: 183.

1,135 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  12. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
38 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  2. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 812
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  4. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  5. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  6. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  7. 2023A Comprehensive Overview of Large Language Modelscited by 639
  8. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  9. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  10. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  11. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  12. 2021MetaICL: Learning to Learn In Contextcited by 450
  13. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  14. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  15. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  16. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  17. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  18. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  19. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  20. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  21. 2024GS-LRM: Large Reconstruction Model for 3D Gaussian Splattingcited by 344
  22. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 320
  23. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  24. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  25. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  26. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generationcited by 258
  27. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 258
  28. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  29. 2021Training Spiking Neural Networks Using Lessons From Deep Learningcited by 251
  30. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  31. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  32. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  33. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  34. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  35. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  36. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  37. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  38. 2022FP8 Formats for Deep Learningcited by 187

50 most cited of 1,135 All 1,135 (JSON)