mlsanityData to 14 Oct 2026

QIGen: Generating Efficient Kernels for Quantized Inference on Large Language Models

arXiv 2307.03738 ↗First posted 7 Jul 2023cs.LG cs.CL cs.PFcited by 6

Not on the map. Why?

Abstract (arXiv)

We present ongoing work on a new automatic code generation approach for supporting quantized generative inference on LLMs such as LLaMA or OPT on off-the-shelf CPUs. Our approach is informed by the target architecture and a performance model, including both hardware characteristics and method-specific accuracy constraints. Results on CPU-based inference for LLaMA models show that our approach can lead to high performance and high accuracy, comparing favorably to the best existing open-source solution. A preliminary implementation is available at https://github.com/IST-DASLab/QIGen.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 11 of its 15 references that are in the map. First posted on arXiv 7 Jul 2023.

Builds on

11 of 15 references.

References · 11

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  9. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  10. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  11. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 2; 2026: 1.

6 papers.

  1. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  2. 2026Statistically-Lossless Quantization of Large Language Modelscited by 6
  3. 2024Dynamic Sentiment Analysis with Local Large Language Models using Majority Voting: A Study on Factors Affecting Restaurant Evaluationcited by 5
  4. 2025How Toxic Can You Get? Search-based Toxicity Testing for Large Language Modelscited by 2
  5. 2025Exploring Large Language Models for Translating Romanian Computational Problems into Englishcited by 2
  6. 2024Evaluating the Performance of Large Language Models in Competitive Programming: A Multi-Year, Multi-Grade Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
  2. 2023Efficient LLM Inference on CPUs
  3. 2024Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs
  4. 2025QuantX: A Framework for Hardware-Aware Quantization of Generative AI Workloads
  5. 2026Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct
  6. 2024Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarks
  7. 2026Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU
  8. 2023QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models