mlsanityData to 14 Oct 2026

Scalify: scale propagation for efficient low-precision LLM training

arXiv 2407.17353 ↗First posted 24 Jul 2024cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Low-precision formats such as float8 have been introduced in machine learning accelerated hardware to improve computational efficiency for large language models training and inference. Nevertheless, adoption by the ML community has been slowed down by the complex, and sometimes brittle, techniques required to match higher precision training accuracy. In this work, we present Scalify, a end-to-end scale propagation paradigm for computational graphs, generalizing and formalizing existing tensor scaling methods. Experiment results show that Scalify supports out-of-the-box float8 matrix multiplication and gradients representation, as well as float16 optimizer state storage. Our JAX implementation of Scalify is open-sourced at https://github.com/graphcore-research/jax-scalify

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 20 of its 31 references that are in the map. First posted on arXiv 24 Jul 2024.

Builds on

20 of 31 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2017Mixed Precision Trainingcited by 1,156
  8. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
  9. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  10. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  11. 2022FP8 Formats for Deep Learningcited by 196
  12. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
8 more
  1. 2023Microscaling Data Formats for Deep Learningcited by 152
  2. 2022FP8 Quantization: The Power of the Exponentcited by 88
  3. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  4. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
  5. 20228-bit Numerical Formats for Deep Neural Networkscited by 25
  6. 2023Unit Scaling: Out-of-the-Box Low-Precision Trainingcited by 17
  7. 2023ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Taskscited by 14
  8. 2024FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Designcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  2. 2025DLP: Dynamic Layerwise Pruning in Large Language Modelscited by 6
  3. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 5
  4. 2025Knowledge Grafting: A Mechanism for Optimizing AI Model Deployment in Resource-Constrained Environmentscited by 0