Beyond Mean Foils: Auditing Worst-Foil Specificity in Frozen CLIP Region Explanations
Not on the map. Why?
Abstract (arXiv)
A region can overlap a target object yet contribute more to another class. We test regions selected by Cluster-based Concept Importance (CCI) in frozen CLIP. Across COCO and VOC with two checkpoints, 41.08-64.78% of regions that pass overlap and mean-contrast checks fail against the strongest competing class. Removing competitors annotated in the image leaves 39.69-63.64% failing. We then test all eight candidate regions per image. An alternative passes the test for 6.25-7.84% of failures on COCO and 27.40-31.15% on VOC. Requiring it to preserve the original target-score drop within ε = 0.02 reduces these rates to 0.16-0.98%. Available regions and target-drop tolerance constrain repair; relaxing the tolerance increases repair opportunities.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 8 of its 14 references that are in the map. First posted on arXiv 23 Sep 2026.
Builds on
8 of 14 references.
References · 8
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
- 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-trainingcited by 96
- 2022Metrics for saliency map evaluation of deep learning explanation methodscited by 40
- 2023Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attributioncited by 34
- 2022Contrastive Corpus Attribution for Explaining Representationscited by 7
- 2025Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approachcited by 3
Built on by
Not cited yet.