mlsanityData to 14 Oct 2026

FidelityQ: Preserving Predictive Fidelity in Ultra-Low-Bit Language Model Quantization

arXiv 2610.14934 ↗First posted 10 Oct 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Ultra-low-bit scalar quantization offers a regular arithmetic structure for efficient deployment of large language models, but preserving predictive behavior at extreme precision remains challenging: with only a few reconstruction levels, accurate weight representation must translate into consistent predictions from a small calibration set. We propose FidelityQ, a scalar quantization framework that preserves predictive fidelity at weight precisions down to 2 bits. For accurate quantization, we introduce Information-Aligned Analytic Coding (IAC), which shapes each weight row toward a centered symmetric source via randomized Hadamard mixing and fits a mirrored four-level minimum-distortion quantizer to its empirical weights. Every level set in this family decomposes exactly into two sign-valued bases with associated scales, combining source-adaptive reconstruction with lookup-free decoding compatible with scaled integer arithmetic. Our analysis further shows that residual predictive discrepancies are heavy-tailed and concentrate on tokens the teacher predicts confidently, motivating Knowledge-Intensive Calibration (KIC) for consistent quantization. KIC ranks sequences by the tail of token-wise teacher-student divergence, allocates a limited calibration budget to the highest-scoring samples, and restores predictive agreement by updating only quantization levels and scales under teacher supervision, keeping pretrained weights frozen. Experiments on the LLaMA and Qwen families demonstrate the effectiveness of FidelityQ in weight-only and joint quantization settings. On LLaMA-2-7B, FidelityQ achieves a WikiText2 perplexity of 6.62 at W2A16 and improves the prior state-of-the-art W2A4KV4 result from 8.31 to 6.96, while reducing linear-layer weight memory by 8.0x relative to FP16.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 47 of its 55 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

47 of 55 references.

References · 47

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
35 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  7. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  9. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  10. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  11. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  12. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  13. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  14. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  15. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  16. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  17. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  18. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  19. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  20. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  21. 2023PB-LLM: Partially Binarized Large Language Modelscited by 83
  22. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 82
  23. 2025GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibrationcited by 60
  24. 2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 58
  25. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 57
  26. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 50
  27. 2024DB-LLM: Accurate Dual-Binarization for Efficient LLMscited by 37
  28. 2024LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Gridcited by 22
  29. 2024Self-calibration for Language Model Quantization and Pruningcited by 13
  30. 2026Rethinking Residual Errors in Compensation-based LLM Quantizationcited by 7
  31. 2026Using large language models for embodied planning introduces systematic safety riskscited by 5
  32. 2025PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recoverycited by 4
  33. 2025Rotate, Clip, and Partition: Towards W2A4KV4 Quantization by Integrating Rotation and Learnable Non-uniform Quantizercited by 4
  34. 2026Qift: Shift-Friendly No-Zero W2 Post-Training Quantization for Rotated W2A4/KV4 LLM Inferencecited by 1
  35. 2026RAG-Stack: Co-Optimizing RAG Serving Performance and Qualitycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Statistically-Lossless Quantization of Large Language Models
  2. 2026Sliced-Wasserstein Distribution Alignment Loss Improves the Ultra-Low-Bit Quantization of Large Language Models
  3. 2025PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models
  4. 2026When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization
  5. 2026BARQ: Balanced Codebook Refinement for Low-Bit LLM Quantization
  6. 2025PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models
  7. 2026On the Expressive Power of Weight Quantization in Large Language Models
  8. 2025Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models