mlsanityData to 14 Oct 2026

Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery

arXiv 2601.20088 ↗First posted 27 Jan 2026cs.LGcited by 22

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

This technical report presents quantization-aware distillation (QAD) and our best practices for recovering accuracy of NVFP4-quantized large language models (LLMs) and vision-language models (VLMs). QAD distills a full-precision teacher model into a quantized student model using a KL divergence loss. While applying distillation to quantized models is not a new idea, we observe key advantages of QAD for today's LLMs: 1. It shows remarkable effectiveness and stability for models trained through multi-stage post-training pipelines, including supervised fine-tuning (SFT), reinforcement learning (RL), and model merging, where traditional quantization-aware training (QAT) suffers from engineering complexity and training instability; 2. It is robust to data quality and coverage, enabling accuracy recovery without full training data. We evaluate QAD across multiple post-trained models including AceReason Nemotron, Nemotron 3 Nano, Nemotron Nano V2, Nemotron Nano V2 VL (VLM), and Llama Nemotron Super v1, showing consistent recovery to near-BF16 accuracy.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 60 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

45 of 60 references.

References · 45

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2019Towards VQA Models That Can Readcited by 1,718
  7. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  12. 2016A Diagram Is Worth A Dozen Imagescited by 804
33 more
  1. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  2. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  3. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  4. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  5. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  6. 2021A White Paper on Neural Network Quantizationcited by 472
  7. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  8. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  9. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  10. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  11. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  12. 2018Model compression via distillation and quantizationcited by 289
  13. 2022FP8 Formats for Deep Learningcited by 196
  14. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
  15. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  16. 2021Towards Understanding Knowledge Distillationcited by 158
  17. 2023Microscaling Data Formats for Deep Learningcited by 152
  18. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  19. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
  20. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
  21. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 101
  22. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  23. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergycited by 77
  24. 2025Pretraining Large Language Models with NVFP4cited by 76
  25. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 69
  26. 2019QKD: Quantization-aware Knowledge Distillationcited by 44
  27. 2025FP4 All the Way: Fully Quantized Training of LLMscited by 40
  28. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationcited by 32
  29. 2020Why distillation helps: a statistical perspectivecited by 21
  30. 2023Token-Scaled Logit Distillation for Ternary Weight Generative Language Modelscited by 20
  31. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 16
  32. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 14
  33. 2024EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 22.

22 papers.

Sort:

By area:

  1. 2026BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimizationcited by 6
  2. 2026Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Languagecited by 4
  3. 2026Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Controlcited by 3
  4. 2026Magpie: Real-Time World Renderer for Interactive Gamescited by 3
  5. 2026Laguna M.1/XS.2 Technical Reportcited by 1
  6. 2026Apertus LLM Family Expansion via Distillation and Quantizationcited by 1
  7. 2026Characterizing the Impact of NVFP4 Quantization for Low-Power Edge AI Deploymentcited by 1
  8. 2026QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstructioncited by 1
  9. 2026QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMscited by 1
  10. 2026Direction-Preserving Number Representationscited by 0
  11. 2026Nano-U: Efficient Terrain Segmentation for Tiny Robot Navigationcited by 0
  12. 2026FTerViT: Fully Ternary Vision Transformercited by 0
10 more
  1. 2026Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillationcited by 0
  2. 2026SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matricescited by 0
  3. 2026Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4Bcited by 0
  4. 2026Studying quantization trade-offs for efficient inference deployment in machine translationcited by 0
  5. 2026Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMscited by 0
  6. 2026Llama-Mobile: Efficient 2.7-Bit Quantization of VLMscited by 0
  7. 2026Disaggregated Quantization: Specializing LLM Prefill and Decodecited by 0
  8. 2026Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inferencecited by 0
  9. 2026LatentQuant: Preserving the Policy-Facing Latent Contract under NVFP4 VAE Quantizationcited by 0
  10. 2026Beyond In-Distribution Preservation: Recovering Generalization in Quantized VLAs via Vulnerability-Oriented Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OnlineQAT: On-Policy Distillation for Ultra-Low-Bit Large Language Models
  2. 2024Oh! We Freeze: Improving Quantized Knowledge Distillation via Signal Propagation Analysis for Large Language Models
  3. 2026QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides
  4. 2026ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
  5. 2026Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs
  6. 2026QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch
  7. 2025Optimizing Large Language Model Training Using FP4 Quantization
  8. 2026Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning