mlsanityData to 14 Oct 2026

Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks

arXiv 1711.02213 ↗First posted 6 Nov 2017cs.LG cs.NA math.NAcited by 44

Not on the map. Why?

Abstract (arXiv)

Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neural networks in low bit-width remains a challenging problem. Here we present the Flexpoint data format, aiming at a complete replacement of 32-bit floating point format training and inference, designed to support modern deep network topologies without modifications. Flexpoint tensors have a shared exponent that is dynamically adjusted to minimize overflows and maximize available dynamic range. We validate Flexpoint by training AlexNet, a deep residual network and a generative adversarial network, using a simulator implemented with the neon deep learning framework. We demonstrate that 16-bit Flexpoint closely matches 32-bit floating point in training all three models, without any need for tuning of model hyperparameters. Our results suggest Flexpoint as a promising numerical format for future hardware for training and inference.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 6; 2023: 9; 2024: 5; 2025: 6; 2026: 3.

44 papers.

Sort:

By area:

  1. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  2. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
  3. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 64
  4. 2021Zero-shot Adversarial Quantizationcited by 44
  5. 2021FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Roundingcited by 31
  6. 2022Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Trainingcited by 26
  7. 2023Data Quality-aware Mixed-precision Quantization via Hybrid Reinforcement Learningcited by 17
  8. 2024CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimizationcited by 15
  9. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  10. 2022FlexBlock: A Flexible DNN Training Accelerator with Multi-Mode Block Floating Point Supportcited by 10
  11. 2021Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniquescited by 9
  12. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 9
32 more
  1. 2024DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analyticscited by 8
  2. 2021All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networkscited by 5
  3. 2023Hierarchical Training of Deep Neural Networks Using Early Exitingcited by 5
  4. 2023Multiplication-Free Transformer Training via Piecewise Affine Operationscited by 5
  5. 2021Rethinking Floating Point Overheads for Mixed Precision DNN Acceleratorscited by 4
  6. 2021Elastic Significant Bit Quantization and Acceleration for Deep Neural Networkscited by 4
  7. 2023Number Systems for Deep Neural Network Architectures: A Surveycited by 4
  8. 2023Tango: rethinking quantization for graph neural network training on GPUscited by 4
  9. 2021The challenge of reproducible ML: an empirical study on the impact of bugscited by 3
  10. 2021MERCURY: Accelerating DNN Training By Exploiting Input Similaritycited by 3
  11. 2022MinUn: Accurate ML Inference on Microcontrollerscited by 3
  12. 2022Accuracy Booster: Enabling 4-bit Fixed-point Arithmetic for DNN Trainingcited by 3
  13. 2025DAF: An Efficient End-to-End Dynamic Activation Framework for on-Device DNN Trainingcited by 3
  14. 2025Low-Precision Streaming PCAcited by 3
  15. 2021Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directionscited by 2
  16. 2024Efficient and Mathematically Robust Operations for Certified Neural Networks Inferencecited by 2
  17. 2021EFloat: Entropy-coded Floating Point Format for Compressing Vector Embedding Modelscited by 1
  18. 2021Representation range needs for 16-bit neural network trainingcited by 1
  19. 2022Schr"odinger's FP: Dynamic Adaptation of Floating-Point Containers for Deep Learning Trainingcited by 1
  20. 2023Ultra-low Precision Multiplication-free Training for Deep Neural Networkscited by 1
  21. 2023Revisiting 16-bit Neural Network Training: A Practical Approach for Resource-Limited Learningcited by 1
  22. 2024Toward INT4 Fixed-Point Training via Exploring Quantization Error for Gradientscited by 1
  23. 2025MAx-DNN: Multi-Level Arithmetic Approximation for Energy-Efficient DNN Hardware Acceleratorscited by 1
  24. 2021A Survey on GAN Acceleration Using Memory Compression Techniquecited by 0
  25. 2022One-way Explainability Isn't The Messagecited by 0
  26. 2023The Hidden Power of Pure 16-bit Floating-Point Neural Networkscited by 0
  27. 2023Compressed Real Numbers for AI: a case-study using a RISC-V CPUcited by 0
  28. 2025Morphing-based Compression for Data-centric ML Pipelinescited by 0
  29. 2025Federated learning over physical channels: adaptive algorithms with near-optimal guaranteescited by 0
  30. 2026Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extendedcited by 0
  31. 2026MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Accelerationcited by 0
  32. 2026LocalTrain: Integer-Only Local Learning for Energy-Efficient CNN Training on Edge FPGAscited by 0