mlsanityData to 14 Oct 2026

Mapping the Schedule x Bit-Width Boundary in Sub-100M Quantisation-Aware Training

arXiv 2605.25966 ↗First posted 25 May 2026cs.LG cs.CL stat.MLcited by 2

Not on the map. Why?

Abstract (arXiv)

We test whether the optimal learning-rate schedule depends on bit-width during from-initialisation quantisation-aware training (QAT) for sub-100M decoder language models. A 720-run factorial grid (Phase 2) over bit-width x warmdown fraction x LR magnitude x model size x seed (FP16/INT8/INT6, 15M-100M, 5 seeds) finds the optimal warmdown is 33% at every (bit-width, size) cell. The primary hypothesis -- that INT6 QAT requires a different schedule than higher-precision training -- is falsified at FP16/INT8/INT6. A 625-run follow-up (Phase 5) probes the null along five axes: optimiser (AdamW), schedule shape (cosine), training length (up to 9x more iterations), an extended size sweep (5M-350M), and an INT4 sweep from 3M to 100M. The null is robust under all three setup changes. The INT6 penalty follows a log-linear scaling law whose fit on Phase 2 predicts the five held-out Phase 5 sizes (5M, 8M, 175M, 250M, 350M) within their 95% prediction intervals (5/5). For INT4 the picture is sharper than the higher precisions: at 50M and 100M, wd33 is decisively optimal (paired z ~ 12-15, 10/10 seeds); below 50M, across the six tested sizes from 3M to 30M, no individual size shows a statistically significant schedule preference and the per-size mean penalty oscillates within seed-level noise. The boundary is therefore a transition between a noise-dominated regime below 50M and a decisive wd33 regime at and above 50M, not a clean wd10 region. A weight-to-grid-distance probe falsifies the simplest mechanism for the FP16/INT8/INT6 null result (rapid grid-snapping): pre-warmdown, INT6-QAT weights sit at essentially the same distance from the INT6 grid as FP16 weights (ratio ~ 1.04). Practical recommendation: at sub-100M scale, tune the LR schedule once at FP16 and apply unchanged to INT8/INT6 QAT; for INT4 at 50M+ use wd33; for INT4 below 50M the schedule choice is in the noise.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 25 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

22 of 25 references.

References · 22

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2019Root Mean Square Layer Normalizationcited by 1,480
  6. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  10. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  11. 2025Muon is Scalable for LLM Trainingcited by 428
  12. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
10 more
  1. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 249
  2. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  3. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  4. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  5. 2024Low-Rank Quantization-Aware Training for LLMscited by 43
  6. 2024BitNet b1.58 Reloaded: State-of-the-art Performance Also on Smaller Networkscited by 8
  7. 2025Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models?cited by 4
  8. 2025Introducing MareNostrum5: A European pre-exascale energy-efficient system designed to serve a broad spectrum of scientific workloadscited by 2
  9. 2025FlexQ: Efficient Post-training INT6 Quantization for LLM Serving via Algorithm-System Co-Designcited by 2
  10. 2025FraQAT: Quantization Aware Training with Fractional bitscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Baseline Shape Decides the Verdict: A Controlled Re-Examination of Ternary Language Models at 60K Parameterscited by 0
  2. 2026Linger and Lose: Knowledge Collapse in Low-Bit Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Compute-Optimal Quantization-Aware Training
  2. 2024Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniques
  3. 2025Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models
  4. 2026Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models
  5. 2026StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths
  6. 2026A Target-Centric Survey of Quantization-Aware Training
  7. 2025Scaling Law for Quantization-Aware Training
  8. 2024Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens