mlsanityData to 7 Oct 2026

The Rescue Effect: Spatio-Semantic Early Exit Bypasses Quantization Collapse in CLIP

arXiv 2605.26415 ↗First posted 26 May 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

also in 1 more
Abstract (arXiv)

Deploying Vision-Language Models on resource-constrained hardware typically requires INT8 quantization, but in joint-embedding architectures such as CLIP this introduces a failure mode distinct from quantized CNN classifiers: activation noise accumulated across transformer blocks perturbs the direction of the multimodal embedding, eroding the cosine alignment on which zero-shot retrieval depends. We characterize this as Quantization-Induced Representation Collapse (QIRC) and quantify it on INT8 CLIP ViT-B/32, where the layer-wise noise-to-signal ratio grows from below 10% in shallow blocks to 52% at Layer 11. We propose LRA-EE (Layer-wise Representation-Aware Early Exit), which bypasses noise-saturated deep layers via Spatio-Semantic Aggregation (replacing the immature shallow [CLS] with a global patch-token average), a learned multi-feature gate (confidence, top-2 margin, spatial-activation variance), and Layer-adaptive Confidence Thresholding calibrated to each layer's Information-to-Noise Ratio. On ImageNet-1K zero-shot classification, LRA-EE reduces FLOPs by 13.4% and improves Top-1 accuracy by +2.44%p (58.72% -> 61.16%) over the INT8 baseline. A four-quadrant decomposition isolates the Rescue Effect: 9.5% of samples are correctly classified at shallow exits but lost to noise at full depth, against only 7.1% suffering the inverse.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 33 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

30 of 33 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  12. 2021Florence: A New Foundation Model for Computer Visioncited by 802
18 more
  1. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  2. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  3. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  5. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  6. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  7. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  8. 2021Post-Training Quantization for Vision Transformercited by 274
  9. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  10. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  11. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  12. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 173
  13. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  14. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  15. 2021Learned Token Pruning for Transformerscited by 138
  16. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
  17. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  18. 2021Multi-Exit Vision Transformer for Dynamic Inferencecited by 28

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Channel-Wise and Token-Aware Post-Training Quantization for Visual State Space Duality
  2. 2025Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracy
  3. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  4. 2026Beyond Reconstruction Loss in Post-Training Quantization: Balanced Fitting for Large Vision-Language Models
  5. 2026What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features
  6. 2025Language-Aware Information Maximization for Transductive Few-Shot CLIP
  7. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference
  8. 2024Quantized Prompt for Efficient Generalization of Vision-Language Models