# The Impact of Quantization on the Robustness of Transformer-based Text Classifiers

arXiv: https://arxiv.org/abs/2403.05365 · HTML: https://mlsanity.com/paper/2403.05365/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 54 references that are in the map.
- First posted on arXiv 8 Mar 2024.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://mlsanity.com/paper/1605.07725/) (2016)
- [LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks](https://mlsanity.com/paper/1807.10029/) (2018)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [Improving Adversarial Robustness via Promoting Ensemble Diversity](https://mlsanity.com/paper/1901.08846/) (2019)
- [Yara Parser: A Fast and Accurate Dependency Parser](https://mlsanity.com/paper/1503.06733/) (2015)
- [Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systems](https://mlsanity.com/paper/1903.11508/) (2019)
- [Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation](https://mlsanity.com/paper/1909.01492/) (2019)
- [Compression of Generative Pre-trained Language Models via Quantization](https://mlsanity.com/paper/2203.10705/) (2022)
- [SAFER: A Structure-free Approach for Certified Robustness to Adversarial Word Substitutions](https://mlsanity.com/paper/2005.14424/) (2020)
- [Robust Quantization: One Model to Rule Them All](https://mlsanity.com/paper/2002.07686/) (2020)
- [Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!](https://mlsanity.com/paper/2103.10013/) (2021)
- [Neural Networks with Few Multiplications](https://mlsanity.com/paper/1510.03009/) (2015)
- [Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs](https://mlsanity.com/paper/2101.10579/) (2021)
- [Robust Machine Comprehension Models via Adversarial Training](https://mlsanity.com/paper/1804.06473/) (2018)
- [Attacking Binarized Neural Networks](https://mlsanity.com/paper/1711.00449/) (2017)
- [EMPIR: Ensembles of Mixed Precision Deep Networks for Increased Robustness against Adversarial Attacks](https://mlsanity.com/paper/2004.10162/) (2020)
- [Automatic Mixed-Precision Quantization Search of BERT](https://mlsanity.com/paper/2112.14938/) (2021)
- [White-to-Black: Efficient Distillation of Black-Box Adversarial Attacks](https://mlsanity.com/paper/1904.02405/) (2019)
- [Discretization based Solutions for Secure Machine Learning against Adversarial Attacks](https://mlsanity.com/paper/1902.03151/) (2019)

## Built on by (most cited first, 2 of 2)

- [Leveraging Generative AI for Enhancing Domain-Driven Software Design](https://mlsanity.com/paper/2601.20909/) (2026)
- [A Hybrid, Multi-Layered Pipeline for Phishing and Threat Classification: Independently Validated URL and NLP Engines with a Calibrated Multi-Channel Fusion Stage](https://mlsanity.com/paper/2606.21690/) (2026)

## Abstract (arXiv)

Transformer-based models have made remarkable advancements in various NLP areas. Nevertheless, these models often exhibit vulnerabilities when confronted with adversarial attacks. In this paper, we explore the effect of quantization on the robustness of Transformer-based models. Quantization usually involves mapping a high-precision real number to a lower-precision value, aiming at reducing the size of the model at hand. To the best of our knowledge, this work is the first application of quantization on the robustness of NLP models. In our experiments, we evaluate the impact of quantization on BERT and DistilBERT models in text classification using SST-2, Emotion, and MR datasets. We also evaluate the performance of these models against TextFooler, PWWS, and PSO adversarial attacks. Our findings show that quantization significantly improves (by an average of 18.68%) the adversarial accuracy of the models. Furthermore, we compare the effect of quantization versus that of the adversarial training approach on robustness. Our experiments indicate that quantization increases the robustness of the model by 18.80% on average compared to adversarial training without imposing any extra computational overhead during training. Therefore, our results highlight the effectiveness of quantization in improving the robustness of NLP models.