mlsanityData to 14 Oct 2026

Quantize with Confidence? An Empirical Study of Quantization for Code Generation

arXiv 2607.14181 ↗First posted 15 Jul 2026cs.SE cs.LG cs.PLcited by 0

Not on the map. Why?

Abstract (arXiv)

The growing adoption of local inference frameworks such as Ollama has made it increasingly common for developers to run large code models on laptops and other resource-constrained hardware. In these settings, post-training quantization is essential for reducing memory footprint and enabling practical deployment, yet its impact on generated code remains insufficiently understood. We empirically evaluate six state-of-the-art quantization methods (GPTQ, AWQ, QuIP#, AQLM, BitsAndBytes, and GGUF) on two representative large code model families, Qwen2.5-Coder and CodeLlama, using the multilingual McEval and CoderEval benchmarks for Python and Java. We assess functional correctness (pass@1) together with maintainability, reliability, security, and structural complexity. We also introduce a novel analysis of robustness under varying prompt complexity, characterized by Shannon entropy and token length. Our results show that quantization techniques differ meaningfully in their impact on correctness and code quality. AQLM consistently matches or exceeds the full-precision baseline, whereas QuIP# exhibits the largest correctness degradation, particularly on complex prompts. Security attributes remain stable across models, benchmarks, and programming languages, while robustness to prompt complexity varies across techniques. These findings provide practical guidance for selecting quantization strategies for deploying large code models on resource-constrained hardware and highlight the importance of evaluating quantized models beyond functional correctness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 70 of its 101 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

70 of 101 references.

References · 70

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
38 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  3. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  4. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  6. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  7. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  8. 2023Large Language Modelscited by 780
  9. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  10. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  11. 2019Learned Step Size Quantizationcited by 628
  12. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  13. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  14. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  15. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  16. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  17. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  18. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  19. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  20. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  21. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  22. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  23. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
  24. 2023A Survey on Model Compression for Large Language Modelscited by 269
  25. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  26. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 181
  27. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 138
  28. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  29. 2022Few-shot training LLMs for project-specific code-summarizationcited by 129
  30. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  31. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  32. 2025CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingscited by 115
  33. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
  34. 2023PB-LLM: Partially Binarized Large Language Modelscited by 83
  35. 2023Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issuescited by 79
  36. 2023Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)cited by 72
  37. 2023Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPTcited by 71
  38. 2023On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilotcited by 64

50 most cited of 70 All 70 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Compressed code: the hidden effects of quantization and distillation on programming tokens
  2. 2024An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2
  3. 2025Smaller = Weaker? Benchmarking Robustness of Quantized LLMs in Code Generation
  4. 2026ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generation
  5. 2024On the Compression of Language Models for Code: An Empirical Study on CodeBERT
  6. 2023Greener yet Powerful: Taming Large Code Generation Models with Quantization
  7. 2025CodeSimpleQA: Scaling Factuality in Code Large Language Models
  8. 2026Code Is More Than Text: Uncertainty Estimation for Code Generation