# SAMP: A Model Inference Toolkit of Post-Training Quantization for Text Processing via Self-Adaptive Mixed-Precision

arXiv: https://arxiv.org/abs/2209.09130 · HTML: https://mlsanity.com/paper/2209.09130/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 19 Sep 2022.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [A Survey on Methods and Theories of Quantized Neural Networks](https://mlsanity.com/paper/1808.04752/) (2018)
- [TurboTransformers: An Efficient GPU Serving System For Transformer Models](https://mlsanity.com/paper/2010.05680/) (2020)
- [FastFormers: Highly Efficient Transformer Models for Natural Language Understanding](https://mlsanity.com/paper/2010.13382/) (2020)

## Built on by (most cited first, 3 of 3)

- [HUMOTO: A 4D Dataset of Mocap Human Object Interactions](https://mlsanity.com/paper/2504.10414/) (2025)
- [Multi-stage Training of Bilingual Islamic LLM for Neural Passage Retrieval](https://mlsanity.com/paper/2501.10175/) (2025)
- [Building an Efficient Multilingual Non-Profit IR System for the Islamic Domain Leveraging Multiprocessing Design in Rust](https://mlsanity.com/paper/2411.06151/) (2024)

## Abstract (arXiv)

The latest industrial inference engines, such as FasterTransformer and TurboTransformers, have verified that half-precision floating point (FP16) and 8-bit integer (INT8) quantization can greatly improve model inference speed. However, the existing INT8 quantization methods are too complicated, and improper usage will lead to model performance damage greatly. In this paper, we develop a toolkit for users to easily quantize their models for inference, in which Self-Adaptive Mixed-Precision (SAMP) is proposed to automatically control quantization rate by a mixed-precision architecture to balance model accuracy and efficiency. Experimental results show that our SAMP toolkit has a higher speedup than PyTorch and FasterTransformer while ensuring the required accuracy. In addition, SAMP is based on a modular design, decoupling the tokenizer, embedding, encoder and target layers, which allows users to handle various downstream tasks and can be seamlessly integrated into PyTorch.