mlsanityData to 7 Oct 2026

Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning

arXiv 2609.26708 ↗First posted 22 Sep 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Quantization-aware distillation (QAD) restores much of the short-form question-answering performance lost to sub-3-bit quantization, yet leaves mathematical and code reasoning substantially impaired. Long generations often degenerate into repetitive loops, exhausting the decoding budget without completing a solution. We trace this gap to quantization-amplified exposure bias: QAD trains on fixed corpus prefixes, while quantization-induced deviations compound along the model's own autoregressive trajectories. To address this mismatch, we introduce an on-policy distillation (OPD) stage that places teacher supervision where the quantized model actually goes. Starting from a QAD checkpoint, the student generates through the quantized forward path used at deployment and receives feedback from a frozen full-precision teacher on its own prefixes, combining dense token-level guidance with task-verifier rewards. Across four models at 2.79 and 1.88 effective bits, OPD raises average BF16 performance retention from 35% to 70% on MATH-500 and from 66% to 91% on HumanEval while preserving short-form performance, with reasoning gains substantially exceeding those of continued teacher-forced QAD in matched-budget comparisons. By coupling QAD's stable low-bit initialization with OPD's on-policy reasoning recovery, our framework provides a comprehensive sub-3-bit solution that preserves broad capabilities while restoring long-form reasoning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 38 references that are in the map. First posted on arXiv 22 Sep 2026.

Builds on

31 of 38 references.

References · 31

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  12. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
19 more
  1. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  2. 2016Sequence-Level Knowledge Distillationcited by 761
  3. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  4. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  5. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  6. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  7. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  8. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  9. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  10. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  11. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  12. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  13. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 115
  14. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 80
  15. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 49
  16. 2025Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deploymentcited by 9
  17. 2026What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studycited by 9
  18. 2026EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillationcited by 2
  19. 2026ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Trainingcited by 2

Built on by

Not cited yet.