Opt4GPTQ: Co-Optimizing Memory and Computation for 4-bit GPTQ Quantized LLM Inference on Heterogeneous Platforms
Not on the map. Why?
Abstract (arXiv)
The increasing adoption of large language models (LLMs) on heterogeneous computing platforms poses significant challenges to achieving high inference efficiency. To address these efficiency bottlenecks across diverse platforms, this paper proposes Opt4GPTQ, a practical optimization method designed for 4-bit GPTQ quantized LLMs inference on heterogeneous AI accelerators. Built upon the vLLM serving system, Opt4GPTQ integrates three platform-level optimization strategies: Shared Memory Buffering Optimization (SMB-Opt), which caches frequently accessed data in shared memory and employs single-threaded writes; Vectorized Memory Loading Optimization (VML-Opt), which utilizes vectorized memory operations for efficient data loading; and Inline Assembly Optimization (ILA-Opt), which directly leverages hardwarenative vector half-precision addition and fused multiply-accumulate instructions. Experimental results show that Opt4GPTQ effectively improves performance across various models while maintaining original model accuracy, achieving throughput gains of up to 84.42%. This work highlights the critical role of platformlevel engineering in enabling efficient LLMs inference on emerging architectures and provides valuable methodologies for future heterogeneous platform adaptation.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 31 references that are in the map. First posted on arXiv 29 Oct 2025.
Builds on
19 of 31 references.
References · 19
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2023Qwen Technical Reportcited by 3,344
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
- 2023Code Llama: Open Foundation Models for Codecited by 2,139
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
- 2023A Survey on Evaluation of Large Language Modelscited by 1,242
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
7 more
- 2023A Survey on Model Compression for Large Language Modelscited by 269
- 2024A Survey on Efficient Inference for Large Language Modelscited by 205
- 2025TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operatorscited by 75
- 2023ChatGPT: A Meta-Analysis after 2.5 Monthscited by 51
- 2024A Comprehensive Study on Quantization Techniques for Large Language Modelscited by 36
- 2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systemscited by 28
- 2025Efficient Architecture for RISC-V Vector Memory Accesscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Efficient LLM Inference on CPUs
- 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
- 2024Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs
- 2026RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems
- 2024LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
- 2025Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency
- 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference
- 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models