mlsanityData to 14 Oct 2026

PACT: Parameterized Clipping Activation for Quantized Neural Networks

arXiv 1805.06085 ↗First posted 16 May 2018cs.CV cs.AIcited by 467

Not on the map. Why?

Abstract (arXiv)

Deep learning algorithms achieve high classification accuracy at the expense of significant computation cost. To address this cost, a number of quantization schemes have been proposed - but most of these techniques focused on quantizing weights, which are relatively smaller in size compared to activations. This paper proposes a novel quantization scheme for activations during training - that enables neural networks to work well with ultra low precision weights and activations without any significant accuracy degradation. This technique, PArameterized Clipping acTivation (PACT), uses an activation clipping parameter α that is optimized during training to find the right quantization scale. PACT allows quantizing activations to arbitrary bit precisions, while achieving much better accuracy relative to published state-of-the-art quantization schemes. We show, for the first time, that both weights and activations can be quantized to 4-bits of precision while still achieving accuracy comparable to full precision networks across a range of popular models and datasets. We also show that exploiting these reduced-precision computational units in hardware can enable a super-linear improvement in inferencing performance due to a significant reduction in the area of accelerator compute engines coupled with the ability to retain the quantized model and activation data in on-chip memories.

Key facts. Cited by 467 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 84; 2022: 89; 2023: 89; 2024: 78; 2025: 76; 2026: 51.

467 papers.

Sort:

By area:

  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  3. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  4. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  5. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  6. 2021Post-Training Quantization for Vision Transformercited by 278
  7. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
  8. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  9. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  10. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 179
  11. 2022On-Device Training Under 256KB Memorycited by 160
  12. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
38 more
  1. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 137
  2. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  3. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  4. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 95
  5. 2024Q-DiT: Accurate Post-Training Quantization for Diffusion Transformerscited by 94
  6. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  7. 2021MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learningcited by 89
  8. 2022Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representationcited by 89
  9. 2022FP8 Quantization: The Power of the Exponentcited by 88
  10. 2021Network Quantization with Element-wise Gradient Scalingcited by 87
  11. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Lengthcited by 85
  12. 2021ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spottingcited by 84
  13. 2023Temporal Dynamic Quantization for Diffusion Modelscited by 81
  14. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 79
  15. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  16. 2022PD-Quant: Post-Training Quantization based on Prediction Difference Metriccited by 73
  17. 2022Deep Differentiable Logic Gate Networkscited by 67
  18. 2024AffineQuant: Affine Transformation Quantization for Large Language Modelscited by 67
  19. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  20. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 65
  21. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 64
  22. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  23. 2021Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimizationcited by 61
  24. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 61
  25. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 61
  26. 2023Training Transformers with 4-bit Integerscited by 60
  27. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 57
  28. 2022Patch Similarity Aware Data-Free Quantization for Vision Transformerscited by 56
  29. 2022Mixed-Precision Neural Network Quantization via Learned Layer-wise Importancecited by 54
  30. 2021Implicit Neural Video Compressioncited by 52
  31. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 52
  32. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 51
  33. 2021PokeBNN: A Binary Pursuit of Lightweight Accuracycited by 49
  34. 2023Towards Accurate Post-training Quantization for Diffusion Modelscited by 47
  35. 2024Q-VLM: Post-training Quantization for Large Vision-Language Modelscited by 47
  36. 2023Pruning vs Quantization: Which is Better?cited by 46
  37. 2022Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Modelscited by 45
  38. 2021Zero-shot Adversarial Quantizationcited by 44

50 most cited of 467 All 467 (JSON)