Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs
Not on the map. Why?
Abstract (arXiv)
Serving large language models cheaply increasingly means shipping models that are both structurally compressed to a fraction of their parameters and quantized to 4 bits. Together these steps degrade reasoning, mathematics, coding, and long-context behavior enough to require a recovery, or healing, stage before deployment. The default recipe, quantization-aware training (QAT), re-fits the compressed, quantized model to hard labels; in our pipeline it converged slowly and collapsed past its peak. We adopted Quantization-Aware Healing (QAH) instead. Because a structurally compressed model is never independently trained at full precision, its bfloat16 checkpoint is a distillation-recovered approximation of the original; QAH distills the 4-bit student directly from the original, uncompressed model. On a GPT-OSS 120B to 60B to MXFP4 pipeline, the QAH student matches or beats its bfloat16 source on 7 of 9 benchmarks at roughly 4 times less weight memory and half the teacher's parameter count, and is released open-weight as Hypernova-60B. Against a matched QAT baseline it reaches a comparable peak about 7 times faster and stays stable under continued training, without hand-tuned early stopping. We also report deployment lessons, including a large, reproducible quality gap between distributed-training backends. Our aim is a recipe deployable without a multi-week hyper-parameter search.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 21 Aug 2026.
Builds on
25 of 38 references.
References · 25
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
- 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
- 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
- 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
- 2018Model compression via distillation and quantizationcited by 287
13 more
- 2015Tensorizing Neural Networkscited by 246
- 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
- 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
- 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
- 2023Microscaling Data Formats for Deep Learningcited by 132
- 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
- 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
- 2026Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillationcited by 53
- 2024Low-Rank Quantization-Aware Training for LLMscited by 42
- 2024CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networkscited by 29
- 2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recoverycited by 20
- 2024Puzzle: Distillation-Based NAS for Inference-Optimized LLMscited by 17
- 2024Efficient Model Compression Techniques with FishLegcited by 4
Built on by
Not cited yet.