mlsanityData to 14 Oct 2026

Gaussian Weight Sampling for Scalable, Efficient and Stable Pseudo-Quantization Training

arXiv 2505.11170 ↗First posted 16 May 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Ever-growing scale of large language models (LLMs) is pushing for improved efficiency, favoring fully quantized training (FQT) over BF16. While FQT accelerates training, it faces consistency challenges and requires searching over an exponential number of cases, each needing over 200B tokens to ensure stability. Pseudo-quantization training (PQT) addresses the issues of FQT, although it is not well-studied. We explore the practical implications of PQT in detail and propose a noise distribution R that is floating-point (FP)-friendly, with ideal properties including stochastic precision annealing. As a result, the proposed method serves as an effective theoretical foundation for low-precision FP parameters through PQT, utilizing efficient fake quantization via an addition and subsequent FP casting. We demonstrate that Gaussian weight sampling is (1) scalable: supports low-precision FP parameters down to FP6 and high-precision noise up to 9-bit with BF16 operator. The proposed method is (2) efficient: incurring computational overhead as low as 1.40% on the A100 GPU in terms of Llama2 training tokens per second, and requiring 2 bytes per parameter in GPU memory. We demonstrate that PQT with Gaussian weight sampling is (3) stable: closely following or even surpassing performance of the BF16 baseline while pre-training GPT2 and Llama2 models with up to 1B parameters and 300B tokens.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 24 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

12 of 24 references.

References · 12

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  4. 2022FP8 Formats for Deep Learningcited by 196
  5. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  6. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 85
  7. 2024Scaling FP8 training to trillion-token LLMscited by 65
  8. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 56
  9. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 54
  10. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 45
  11. 2025Oscillation-Reduced MXFP4 Training for Vision Transformerscited by 21
  12. 2020Romu: Fast Nonlinear Pseudo-Random Number Generators Providing High Qualitycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Direct Quantized Training of Language Models with Stochastic Rounding
  2. 2025FP4 All the Way: Fully Quantized Training of LLMs
  3. 2023QuIP: 2-Bit Quantization of Large Language Models With Guarantees
  4. 2026D2Quant: Accurate Low-bit Post-Training Weight Quantization for LLMs
  5. 2025Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment
  6. 2026SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models
  7. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models
  8. 2025DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models