mlsanityData to 7 Oct 2026

From Bits to Chips: An LLM-based Hardware-Aware Quantization Agent for Streamlined Deployment of LLMs

arXiv 2601.03484 ↗First posted 7 Jan 2026cs.LGcited by 1

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Deploying models, especially large language models (LLMs), is becoming increasingly attractive to a broader user base, including those without specialized expertise. However, due to the resource constraints of certain hardware, maintaining high accuracy with larger model while meeting the hardware requirements remains a significant challenge. Model quantization technique helps mitigate memory and compute bottlenecks, yet the added complexities of tuning and deploying quantized models further exacerbates these challenges, making the process unfriendly to most of the users. We introduce the Hardware-Aware Quantization Agent (HAQA), an automated framework that leverages LLMs to streamline the entire quantization and deployment process by enabling efficient hyperparameter tuning and hardware configuration, thereby simultaneously improving deployment quality and ease of use for a broad range of users. Our results demonstrate up to a 2.3x speedup in inference, along with increased throughput and improved accuracy compared to unoptimized models on Llama. Additionally, HAQA is designed to implement adaptive quantization strategies across diverse hardware platforms, as it automatically finds optimal settings even when they appear counterintuitive, thereby reducing extensive manual effort and demonstrating superior adaptability. Code will be released.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 41 references that are in the map. First posted on arXiv 7 Jan 2026.

Builds on

30 of 41 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  10. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
18 more
  1. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  2. 2023Large Language Models as Optimizerscited by 830
  3. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  4. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  5. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  6. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  7. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  8. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 427
  9. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  10. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
  11. 2021Post-Training Quantization for Vision Transformercited by 274
  12. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  13. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 113
  14. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  15. 2023AutoML-GPT: Automatic Machine Learning with GPTcited by 80
  16. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  17. 2023MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Taskscited by 66
  18. 2024Large Language Model Agent for Hyper-Parameter Optimizationcited by 62

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026LoKA: Low-precision Kernel Applications for Recommendation Models At Scalecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023A Speed Odyssey for Deployable Quantization of LLMs
  2. 2023Efficient LLM Inference on CPUs
  3. 2026QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs
  4. 2026RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference
  5. 2025Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency
  6. 2026ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs
  7. 2024LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
  8. 2025On-Device Qwen2.5: Efficient LLM Inference with Model Compression and Hardware Acceleration