mlsanityData to 7 Oct 2026

Activation Quantization of Vision Encoders Needs Prefixing Registers

arXiv 2510.04547 ↗First posted 6 Oct 2025cs.LG cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Large pretrained vision encoders are central to multimodal intelligence, powering applications from on-device vision processing to vision-language models. Since these applications often demand real-time processing of massive visual data, reducing the inference cost of vision encoders is critical. Quantization offers a practical path, but it remains challenging even at 8-bit precision due to so-called outliers. In this work, we propose textitRegCache, a training-free algorithm that mitigates outliers in large-scale pretrained vision encoders and serves as a plug-in module that can be applied on top of other quantization methods. RegCache introduces outlier-prone yet semantically meaningless prefix tokens to the vision encoder, which prevent other tokens from having outliers. Notably, we observe that outliers in vision encoders behave differently from those in language models, motivating two technical innovations: middle-layer prefixing and token deletion. Experimental results show that our method consistently improves quantized model performance across various vision encoders, particularly in extremely low-bit regimes (e.g., 4-bit).

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 66 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

51 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Models5 Mar
  2. 2024FastVLM: Efficient Vision Encoding for Vision Language Models17 Dec

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2013Describing Textures in the Wildcited by 2,110
37 more
  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  3. 2025DINOv3cited by 1,297
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  5. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  6. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  7. 2023Vision Transformers Need Registerscited by 844
  8. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  9. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  10. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  11. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  12. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
  13. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  14. 2024Massive Activations in Large Language Modelscited by 277
  15. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  16. 2021Post-Training Quantization for Vision Transformercited by 274
  17. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  18. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 185
  19. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 173
  20. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  21. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  22. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 134
  23. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  24. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 110
  25. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matterscited by 73
  26. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 59
  27. 2025Vision Transformers Don't Need Trained Registerscited by 39
  28. 2024Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantizationcited by 33
  29. 2024Towards Accurate Post-Training Quantization of Vision Transformers via Error Reductioncited by 23
  30. 2024Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMscited by 21
  31. 2025MobileCLIP2: Improving Multi-Modal Reinforced Trainingcited by 20
  32. 2024DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformerscited by 19
  33. 2025APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformerscited by 17
  34. 2025FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximationcited by 16
  35. 2024PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantizationcited by 11
  36. 2024MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformercited by 10
  37. 2025Artifacts and Attention Sinks: Structured Approximations for Efficient Vision Transformerscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Joint Quantization and Token Pruning of Vision-Language Models
  2. 2026Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients
  3. 2026QAPruner: Quantization-Aware Vision Token Pruning for Multimodal Large Language Models
  4. 2024Q-VLM: Post-training Quantization for Large Vision-Language Models
  5. 2026Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization
  6. 2026Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models
  7. 2026SPHQuant: Efficient extreme low bit weight quantization for Vision-Language Models
  8. 2025Variation-aware Vision Token Dropping for Faster Large Vision-Language Models