mlsanityData to 7 Oct 2026

When Masking Helps or Hurts Robustness in Compressed CLIP: A Pre-Deployment Diagnostic

arXiv 2609.39704 ↗First posted 30 Sep 2026cs.CV cs.AIcited by 0

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

This paper demonstrate that whether masking-based token pruning helps or hurts worst-group robustness can be predicted before deployment, without labels or fine-tuning. A systematic study of semantic masking across 8 spurious-correlation benchmarks shows its effect on worst-group accuracy is highly unstable: it improves accuracy by up to 82.5% relative on some datasets and degrades it by up to 100% on others. We trace this instability to spurious inversion: background patches receive higher CLIP text-similarity than the true object when the spurious attribute is background-separable, inverting the assumption every text- and attention-guided pruning method relies on. We introduce the Spurious Inversion Metric (SIM), a label-free, pre-deployment diagnostic whose sign predicts this effect with statistical significance (binomial p=0.035) across all 8 datasets, and remains dependable across 6 CLIP architectures with a clean foreground/background split. Naive masking is itself a major source of risk: it causes the largest average-accuracy loss of any method we evaluate, and its own per-image segmentation step is a significant runtime bottleneck. To address this, we design a batched, synchronization-free GPU segmentation routine that cuts this overhead from 3.5× to 1.75× baseline. Gating deployment by SIM's sign recovers masking's benefits while avoiding its worst failures, matching or exceeding a strong pruning baseline on 7 of 8 datasets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 19 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

17 of 19 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Efficient Vision-Language Reasoning via Adaptive Token Pruning14 Dec
  2. 2025PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models11 Apr
  3. 2024Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration26 Nov
  4. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 13

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  3. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  4. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  5. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  6. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
  7. 2022A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Otherscited by 85
  8. 2022MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflictscited by 81
  9. 2024The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learningcited by 39
  10. 2024ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Objectcited by 22
  11. 2025Debiasing CLIP: Interpreting and Correcting Bias in Attention Headscited by 4
  12. 2024Patch Ranking: Efficient CLIP by Learning to Rank Local Patchescited by 3
1 more
  1. 2025SAGE: Spuriousness-Aware Guided Prompt Exploration for Mitigating Multimodal Biascited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity
  2. 2025Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods
  3. 2023Understanding the Vulnerability of CLIP to Image Compression
  4. 2026Less Is More -- Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Models
  5. 2024Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations
  6. 2026On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression
  7. 2024A Sober Look at the Robustness of CLIPs to Spurious Features
  8. 2025Grounding-Aware Token Pruning: Recovering from Drastic Performance Drops in Visual Grounding Caused by Pruning