# Power-of-Two Quantization-Aware-Training (PoT-QAT) in Large Language Models (LLMs)

arXiv: https://arxiv.org/abs/2601.02298 · HTML: https://mlsanity.com/paper/2601.02298/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 2 of its 2 references that are in the map.
- First posted on arXiv 5 Jan 2026.

## Builds on (most cited first, 2 of 2)

- [Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation](https://mlsanity.com/paper/2004.09602/) (2020)
- [Power-of-Two Quantization for Low Bitwidth and Hardware Compliant Neural Networks](https://mlsanity.com/paper/2203.05025/) (2022)

## Abstract (arXiv)

In Large Language Models (LLMs), the number of parameters has grown exponentially in the past few years, e.g., from 1.5 billion parameters in GPT-2 to 175 billion in GPT-3 to possibly more than trillion in higher versions. This raises a significant challenge for implementation, especially for Edge devices. Unlike cloud computing, memory and processing power for Edge devices are very limited, which necessitates developing novel ideas to make such applications feasible. In this work, we investigate compressing weights with a special quantization that limits numbers to only power-of-two (PoT). This helps save a huge amount of memory as only exponents need to be stored, more importantly, it significantly reduces processing power by replacing costly multiplication with low cost bit shifting. To overcome performance loss due to this strict quantization, we investigate Quantization Aware Training (QAT) to enhance performance through additional training. Results on GPT-2 124M show a major enhancement for quantized PoT model after additional training, with a perplexity enhancement of 66% and BERT-Score loss to baseline GPT-2 of 1%. The memory saving is estimated to be 87.5% while the inference speed is expected to be 3-10x faster with PoT quantization versus full-precision.