# Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks

arXiv: https://arxiv.org/abs/2001.05674 · HTML: https://mlsanity.com/paper/2001.05674/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jan 2020.

## Built on by (most cited first, 23 of 23)

- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [Stable and low-precision training for large-scale vision-language models](https://mlsanity.com/paper/2304.13013/) (2023)
- [ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats](https://mlsanity.com/paper/2307.09782/) (2023)
- [A Survey on Efficient Training of Transformers](https://mlsanity.com/paper/2302.01107/) (2023)
- [FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs](https://mlsanity.com/paper/2309.01172/) (2023)
- [ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks](https://mlsanity.com/paper/2312.08583/) (2023)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression](https://mlsanity.com/paper/2410.12707/) (2024)
- [All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networks](https://mlsanity.com/paper/2104.07329/) (2021)
- [Hierarchical Training of Deep Neural Networks Using Early Exiting](https://mlsanity.com/paper/2303.02384/) (2023)
- [Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators](https://mlsanity.com/paper/2101.11748/) (2021)
- [Denoising Noisy Neural Networks: A Bayesian Approach with Compensation](https://mlsanity.com/paper/2105.10699/) (2021)
- [FP8-BERT: Post-Training Quantization for Transformer](https://mlsanity.com/paper/2312.05725/) (2023)
- [Training with Mixed-Precision Floating-Point Assignments](https://mlsanity.com/paper/2301.13464/) (2023)
- [Gated Compression Layers for Efficient Always-On Models](https://mlsanity.com/paper/2303.08970/) (2023)
- [Towards Accurate and Efficient Sub-8-Bit Integer Training](https://mlsanity.com/paper/2411.10948/) (2024)
- [EFloat: Entropy-coded Floating Point Format for Compressing Vector Embedding Models](https://mlsanity.com/paper/2102.02705/) (2021)
- [Ultra-low Precision Multiplication-free Training for Deep Neural Networks](https://mlsanity.com/paper/2302.14458/) (2023)
- [Toward INT4 Fixed-Point Training via Exploring Quantization Error for Gradients](https://mlsanity.com/paper/2407.12637/) (2024)
- [LightNorm: Area and Energy-Efficient Batch Normalization Hardware for On-Device DNN Training](https://mlsanity.com/paper/2211.02686/) (2022)
- [Why Does Stochastic Gradient Descent Slow Down in Low-Precision Training?](https://mlsanity.com/paper/2508.07142/) (2025)

## Abstract (arXiv)

Training with larger number of parameters while keeping fast iterations is an increasingly adopted strategy and trend for developing better performing Deep Neural Network (DNN) models. This necessitates increased memory footprint and computational requirements for training. Here we introduce a novel methodology for training deep neural networks using 8-bit floating point (FP8) numbers. Reduced bit precision allows for a larger effective memory and increased computational speed. We name this method Shifted and Squeezed FP8 (S2FP8). We show that, unlike previous 8-bit precision training methods, the proposed method works out-of-the-box for representative models: ResNet-50, Transformer and NCF. The method can maintain model accuracy without requiring fine-tuning loss scaling parameters or keeping certain layers in single precision. We introduce two learnable statistics of the DNN tensors - shifted and squeezed factors that are used to optimally adjust the range of the tensors in 8-bits, thus minimizing the loss in information due to quantization.