mlsanityData to 14 Oct 2026

Does Marginal Coverage Guarantee Class-Conditional Safety for Zero-Shot VLMs Under Shift?

arXiv 2608.19376 ↗First posted 19 Aug 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Split-conformal prediction provides marginal coverage under exchangeability and is increasingly used as an abstention layer for zero-shot vision-language models (VLMs). We audit this practice under deployment shift for CLIP, OpenCLIP, and SigLIP across ImageNet and non-ImageNet settings. Marginal coverage can remain relatively high while class-conditional tail coverage collapses: on ImageNet-Sketch, worst-class coverage falls to ≈ 0 and 10-12% of classes lie below a finite-sample null floor, despite marginal coverage of about 0.86. The failure is aligned with target-domain class accuracy but is not predicted by the source-domain diagnostics we test. Source-side Mondrian calibration improves the in-distribution tail but does not transfer, while clustered conformal and Conf-OT improve marginal or average metrics without recovering the worst-class tail. Target-side class calibration substantially lifts the tail, but requires labels for every class and remains set-size-intensive. We further identify a 2-3× cross-family efficiency gap and show that native SigLIP sigmoid scores remove APS's probability-mass interpretation. The findings persist across the tested model scale, pretraining corpus, prompt, miscoverage level α, and shifted non-ImageNet settings. Marginal conformal coverage should therefore be treated as an average reliability statistic, not as a safety guarantee for the class tail.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 45 references that are in the map. First posted on arXiv 19 Aug 2026.

Builds on

33 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Conformal Prediction for Zero-Shot Models30 May

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  7. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  8. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  9. 2017Selective Classification for Deep Neural Networkscited by 852
  10. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  12. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
20 more
  1. 2019Conformal Prediction Under Covariate Shiftcited by 519
  2. 2020Classification with Valid and Adaptive Coveragecited by 396
  3. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 375
  4. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 366
  5. 2016Least Ambiguous Set-Valued Classifiers with Bounded Error Levelscited by 272
  6. 2023Class-Conditional Conformal Prediction with Many Classescited by 109
  7. 2024Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibrationcited by 19
  8. 2024Are foundation models for computer vision good conformal predictors?cited by 13
  9. 2025Kandinsky Conformal Prediction: Beyond Class- and Covariate-Conditional Coveragecited by 13
  10. 2023Empirically Validating Conformal Prediction on Modern Vision Architectures Under Distribution Shift and Long-tailed Datacited by 8
  11. 2025Trustworthy Few-Shot Transfer of Medical VLMs through Split Conformal Predictioncited by 8
  12. 2025Conditional Coverage Diagnostics for Conformal Predictioncited by 8
  13. 2025Conformal Prediction for Long-Tailed Classificationcited by 7
  14. 2024Adapting Prediction Sets to Distribution Shifts Without Labelscited by 6
  15. 2023An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibrationcited by 5
  16. 2025WQLCP: Weighted Adaptive Conformal Prediction for Robust Uncertainty Quantification Under Distribution Shiftscited by 5
  17. 2025Conformal Prediction Meets Long-tail Classificationcited by 5
  18. 2025Conformal Inference for Open-Set and Imbalanced Classificationcited by 3
  19. 2026LATA: Laplacian-Assisted Transductive Adaptation for Conformal Uncertainty in Medical VLMscited by 2
  20. 2026Audited Conformal Prediction for Classification under Unknown Distribution Shiftcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Mitigating Class-Tail Undercoverage in Medical Vision-Language Models under Clinical Shift
  2. 2026Localized Conformal Prediction for Image Classification with Vision-Language Models
  3. 2026PRiSM: Prototype Regularization for Few-Shot VLMs
  4. 2025To Trust Or Not To Trust Your Vision-Language Model's Prediction
  5. 2026DatBench: Discriminative, Faithful, and Efficient VLM Evaluations
  6. 2026Underrepresented in Foundation Model Pretraining Data? A One-Shot Probe
  7. 2024An Empirical Study Into What Matters for Calibrating Vision-Language Models
  8. 2026Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries