mlsanityData to 14 Oct 2026

QuantX: A Framework for Hardware-Aware Quantization of Generative AI Workloads

arXiv 2505.07531 ↗First posted 12 May 2025cs.AI eess.SPcited by 0

Not on the map. Why?

Abstract (arXiv)

We present QuantX: a tailored suite of recipes for LLM and VLM quantization. It is capable of quantizing down to 3-bit resolutions with minimal loss in performance. The quantization strategies in QuantX take into account hardware-specific constraints to achieve efficient dequantization during inference ensuring flexible trade-off between runtime speed, memory requirement and model accuracy. Our results demonstrate that QuantX achieves performance within 6% of the unquantized model for LlaVa-v1.6 quantized down to 3-bits for multiple end user tasks and outperforms recently published state-of-the-art quantization techniques. We further integrate one particular technique from QuantX into the popular Llama.cpp framework and show its feasibility in terms of runtime compared to the mainstream quantization techniques from Llama.cpp. Lastly, this manuscript provides insights into the LLM quantization process that motivated the range of recipes and options that are incorporated in QuantX.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 7 of its 8 references that are in the map. First posted on arXiv 12 May 2025.

Builds on

7 of 8 references.

References · 7

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  6. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  7. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Bits to Chips: An LLM-based Hardware-Aware Quantization Agent for Streamlined Deployment of LLMs
  2. 2026QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs
  3. 2025LLMPi: Optimizing LLMs for High-Throughput on Raspberry Pi
  4. 2023A Speed Odyssey for Deployable Quantization of LLMs
  5. 2023QIGen: Generating Efficient Kernels for Quantized Inference on Large Language Models
  6. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference
  7. 2024A Comprehensive Study on Quantization Techniques for Large Language Models
  8. 20261-Bit Wonder: Improving QAT Performance in the Low-Bit Regime through K-Means Quantization