mlsanityData to 14 Oct 2026

SONIQ: System-Optimized Noise-Injected Ultra-Low-Precision Quantization with Full-Precision Parity

arXiv 2311.14114 ↗First posted 23 Nov 2023cs.AR cs.LG cs.PFcited by 0

Not on the map. Why?

Abstract (arXiv)

Ultra-low-precision inference can sharply reduce memory and latency but often degrades accuracy and relies on specialized hardware. We present SONIQ, a system-optimized, noise-injected quantization framework that learns per-channel mixed precision for both weights and activations while training under the same rules used at inference. By injecting hardware-calibrated quantization noise during training, SONIQ steers models toward the discrete arithmetic used at deployment -- without bespoke runtimes. Across CNNs and Transformers, SONIQ achieves up to 16x and 7x compression, respectively, while matching or exceeding full-precision accuracy. Measured end-to-end, SONIQ delivers up to 7.3x CPU speedup over strong INT8 baselines and up to 6.3x (vector units) / 2.8x (tensor cores) GPU speedup relative to FP16. A practical outcome is that two per-channel precision levels -- one in the 1--4-bit range and one in the 4--8-bit range -- suffice in practice; at inference, each channel selects one of the two, keeping kernels simple and fast. To our knowledge, SONIQ is the first framework to reach or surpass full-precision accuracy under ultra-low (1--4 bits per parameter) regimes while remaining deployable on commodity hardware, narrowing the gap between quantization theory and practical, high-throughput inference.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 94 references that are in the map. First posted on arXiv 23 Nov 2023.

Builds on

42 of 94 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2016Densely Connected Convolutional Networkscited by 6,720
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  9. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  10. 2022Diffusion Models in Vision: A Surveycited by 1,015
  11. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  12. 2019Learned Step Size Quantizationcited by 628
30 more
  1. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  2. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  3. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  4. 2021A White Paper on Neural Network Quantizationcited by 472
  5. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  6. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  7. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  8. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  9. 2017Noisy Networks for Explorationcited by 266
  10. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
  11. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  12. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  13. 2020Characterising Bias in Compressed Modelscited by 167
  14. 2022On-Device Training Under 256KB Memorycited by 160
  15. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  16. 2017Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networkscited by 109
  17. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  18. 2022Mixed-Precision Neural Network Quantization via Learned Layer-wise Importancecited by 54
  19. 2021BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantizationcited by 39
  20. 2018Optimizing CNN Model Inference on CPUscited by 29
  21. 2018BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weightscited by 24
  22. 2023Q-DETR: An Efficient Low-Bit Quantized Detection Transformercited by 24
  23. 2023EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantizationcited by 22
  24. 2022Dynamic Dual Trainable Bounds for Ultra-low Precision Super-Resolution Networkscited by 21
  25. 2022Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Modelscited by 21
  26. 2020MLPerf Mobile Inference Benchmarkcited by 14
  27. 2022RecD: Deduplication for End-to-End Deep Learning Recommendation Model Training Infrastructurecited by 9
  28. 2020Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networkscited by 7
  29. 2023Ultra-low Precision Multiplication-free Training for Deep Neural Networkscited by 1
  30. 2023YFlows: Systematic Dataflow Exploration and Code Generation for Efficient Neural Network Inference using SIMD Architectures on CPUscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
  2. 2025Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference
  3. 2026Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training
  4. 2026StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths
  5. 2025FlexiQ: Adaptive Mixed-Precision Quantization for Latency/Accuracy Trade-Offs in Deep Neural Networks
  6. 2026Beyond Outliers: A Data-Free Layer-wise Mixed-Precision Quantization Approach Driven by Numerical and Structural Dual-Sensitivity
  7. 2026SPARQLe: Sub-Precision Activation Representation for Quantized LLM Inference
  8. 2023On-Chip Hardware-Aware Quantization for Mixed Precision Neural Networks