# Beyond Mean Foils: Auditing Worst-Foil Specificity in Frozen CLIP Region Explanations

arXiv: https://arxiv.org/abs/2609.27356 · HTML: https://mlsanity.com/paper/2609.27356/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 14 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 8 of 8)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [A Closer Look at the Explainability of Contrastive Language-Image Pre-training](https://mlsanity.com/paper/2304.05653/) (2023)
- [Metrics for saliency map evaluation of deep learning explanation methods](https://mlsanity.com/paper/2201.13291/) (2022)
- [Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attribution](https://mlsanity.com/paper/2312.17174/) (2023)
- [Contrastive Corpus Attribution for Explaining Representations](https://mlsanity.com/paper/2210.00107/) (2022)
- [Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approach](https://mlsanity.com/paper/2511.12978/) (2025)

## Abstract (arXiv)

A region can overlap a target object yet contribute more to another class. We test regions selected by Cluster-based Concept Importance (CCI) in frozen CLIP. Across COCO and VOC with two checkpoints, 41.08-64.78% of regions that pass overlap and mean-contrast checks fail against the strongest competing class. Removing competitors annotated in the image leaves 39.69-63.64% failing. We then test all eight candidate regions per image. An alternative passes the test for 6.25-7.84% of failures on COCO and 27.40-31.15% on VOC. Requiring it to preserve the original target-score drop within ε = 0.02 reduces these rates to 0.16-0.98%. Available regions and target-drop tolerance constrain repair; relaxing the tolerance increases repair opportunities.