# MixA-Q: Revisiting Activation Sparsity for Vision Transformers from a Mixed-Precision Quantization Perspective

arXiv: https://arxiv.org/abs/2507.19131 · HTML: https://mlsanity.com/paper/2507.19131/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 36 references that are in the map.
- First posted on arXiv 25 Jul 2025.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision](https://mlsanity.com/paper/1905.03696/) (2019)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [Data-Free Quantization Through Weight Equalization and Bias Correction](https://mlsanity.com/paper/1906.04721/) (2019)
- [HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks](https://mlsanity.com/paper/1911.03852/) (2019)
- [K-Net: Towards Unified Image Segmentation](https://mlsanity.com/paper/2106.14855/) (2021)
- [pymoo: Multi-objective Optimization in Python](https://mlsanity.com/paper/2002.04504/) (2020)
- [Adaptive Token Sampling For Efficient Vision Transformers](https://mlsanity.com/paper/2111.15667/) (2021)
- [SPViT: Enabling Faster Vision Transformers via Soft Token Pruning](https://mlsanity.com/paper/2112.13890/) (2021)
- [RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers](https://mlsanity.com/paper/2212.08254/) (2022)
- [Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer](https://mlsanity.com/paper/2210.06707/) (2022)
- [APQ: Joint Search for Network Architecture, Pruning and Quantization Policy](https://mlsanity.com/paper/2006.08509/) (2020)
- [QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models](https://mlsanity.com/paper/2310.08041/) (2023)
- [SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer](https://mlsanity.com/paper/2303.17605/) (2023)
- [Oscillation-free Quantization for Low-bit Vision Transformers](https://mlsanity.com/paper/2302.02210/) (2023)
- [Revisiting Token Pruning for Object Detection and Instance Segmentation](https://mlsanity.com/paper/2306.07050/) (2023)
- [OMPQ: Orthogonal Mixed Precision Quantization](https://mlsanity.com/paper/2109.07865/) (2021)
- [COCO-O: A Benchmark for Object Detectors under Natural Distribution Shifts](https://mlsanity.com/paper/2307.12730/) (2023)
- [Towards Accurate Post-Training Quantization of Vision Transformers via Error Reduction](https://mlsanity.com/paper/2407.06794/) (2024)
- [EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantization](https://mlsanity.com/paper/2307.10554/) (2023)
- [Jumping through Local Minima: Quantization in the Loss Landscape of Vision Transformers](https://mlsanity.com/paper/2308.10814/) (2023)
- [Patch-wise Mixed-Precision Quantization of Vision Transformer](https://mlsanity.com/paper/2305.06559/) (2023)
- [MetaMix: Meta-state Precision Searcher for Mixed-precision Activation Quantization](https://mlsanity.com/paper/2311.06798/) (2023)

## Built on by (most cited first, 3 of 3)

- [HTTM: Head-wise Temporal Token Merging for Faster VGGT](https://mlsanity.com/paper/2511.21317/) (2025)
- [OFBD: Object-Focused Background Debiasing for Long-Tailed Learning](https://mlsanity.com/paper/2609.37331/) (2026)
- [Hardware-aware Calibrated Clustered Attention for Efficient Visual Geometric Transformers](https://mlsanity.com/paper/2610.09274/) (2026)

## Abstract (arXiv)

In this paper, we propose MixA-Q, a mixed-precision activation quantization framework that leverages intra-layer activation sparsity (a concept widely explored in activation pruning methods) for efficient inference of quantized window-based vision transformers. For a given uniform-bit quantization configuration, MixA-Q separates the batched window computations within Swin blocks and assigns a lower bit width to the activations of less important windows, improving the trade-off between model performance and efficiency. We introduce a Two-Branch Swin Block that processes activations separately in high- and low-bit precision, enabling seamless integration of our method with most quantization-aware training (QAT) and post-training quantization (PTQ) methods, or with simple modifications. Our experimental evaluations over the COCO dataset demonstrate that MixA-Q achieves a training-free 1.35x computational speedup without accuracy loss in PTQ configuration. With QAT, MixA-Q achieves a lossless 1.25x speedup and a 1.53x speedup with only a 1% mAP drop by incorporating activation pruning. Notably, by reducing the quantization error in important regions, our sparsity-aware quantization adaptation improves the mAP of the quantized W4A4 model (with both weights and activations in 4-bit precision) by 0.7%, reducing quantization degradation by 24%.