# Evaluating the Impact of Post-Training Quantization on Large Language Models for Code Generation

arXiv: https://arxiv.org/abs/2503.07103 · HTML: https://mlsanity.com/paper/2503.07103/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2025.

## Built on by (most cited first, 3 of 3)

- [Quantitative Analysis of Performance Drop in DeepSeek Model Quantization](https://mlsanity.com/paper/2505.02390/) (2025)
- [Inference Economics of Enterprise Coding Agents: A Case Study of Cloud vs. On-Premise LLMs](https://mlsanity.com/paper/2607.13080/) (2026)
- [Lightweight but Linguistically Fair? Quantisation Robustness across Bavarian, German, and English](https://mlsanity.com/paper/2610.18343/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown an impressive capability in code generation. The LLM effectiveness generally increases with its size: The higher the number of LLM's trainable parameters the better its ability to implement code. However, when it comes to deploying LLM-based code generators, larger LLMs pose significant challenges related to their memory (and, consequently, carbon) footprint. A previous work by Wei et al. proposed to leverage quantization techniques to reduce the memory footprint of LLM-based code generators without substantially degrading their effectiveness. In short, they studied LLMs featuring up to 16B parameters, quantizing their precision from floating point 32 bits down to int 8 bits and showing their limited impact on code generation performance. Given the fast pace at which LLM capabilities and quantization techniques are evolving, in this work we present a differentiated replication of the work by Wei et al. in which we consider (i) on the one side, more recent and larger code-related LLMs, of up to 34B parameters; (ii) the latest advancements in model quantization techniques, which allow pushing the compression to the extreme quantization level of 2 bits per model parameter and; (iii) different types of calibration datasets to guide the quantization process, including code-specific ones. Our empirical evaluation reveals that the new frontier for LLM quantization is 4-bit precision, resulting in an average memory footprint reduction of 70% compared to the original model without observing any significant decrease in performance. Additionally, when the quantization becomes even more extreme (3 and 2 bits), a code-specific calibration dataset helps to limit the loss of performance.