mlsanityData to 7 Oct 2026

Delta Energy: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization

arXiv 2510.11296 ↗First posted 13 Oct 2025cs.CV cs.LGcited by 1

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent approaches for vision-language models (VLMs) have shown remarkable success in achieving fast downstream adaptation. When applied to real-world downstream tasks, VLMs inevitably encounter both the in-distribution (ID) data and out-of-distribution (OOD) data. The OOD datasets often include both covariate shifts (e.g., known classes with changes in image styles) and semantic shifts (e.g., test-time unseen classes). This highlights the importance of improving VLMs' generalization ability to covariate-shifted OOD data, while effectively detecting open-set semantic-shifted OOD classes. In this paper, inspired by the substantial energy change observed in closed-set data when re-aligning vision-language modalities (specifically by directly reducing the maximum cosine similarity to a low value), we introduce a novel OOD score, named DeltaEnergy. DeltaEnergy significantly outperforms the vanilla energy-based OOD score and provides a more reliable approach for OOD detection. Furthermore, DeltaEnergy can simultaneously improve OOD generalization under covariate shifts, which is achieved by lower-bound maximization for DeltaEnergy (termed EBM). EBM is theoretically proven to not only enhance OOD detection but also yields a domain-consistent Hessian, which serves as a strong indicator for OOD generalization. Based on this finding, we developed a unified fine-tuning framework that allows for improving VLMs' robustness in both OOD generalization and OOD detection. Extensive experiments on challenging OOD detection and generalization benchmarks demonstrate the superiority of our method, outperforming recent approaches by 10% to 25% in AUROC.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 73 of its 94 references that are in the map. First posted on arXiv 13 Oct 2025.

Builds on

73 of 94 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP31 Oct
  2. 2024AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Models26 Oct
  3. 2024Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models11 Oct
  4. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey31 Jul
  5. 2024LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models12 Jul
  6. 2024GalLoP: Learning Global and Local Prompts for Vision-Language Models1 Jul
  7. 2024DeCoOp: Robust Prompt Tuning with Out-of-Distribution Detection1 Jun
  8. 2024CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection26 May
  9. 2024Negative Label Guided OOD Detection with Pretrained Vision-Language Models29 Mar
  10. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Models5 Mar
  11. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Model7 Nov
  12. 2023Unsupervised Prototype Adapter for Vision-Language Models22 Aug
8 more
  1. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Prompts14 Jul
  2. 2023CoPL: Contextual Prompt Learning for Vision-Language Understanding3 Jul
  3. 2023How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?9 Jun
  4. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  7. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  8. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  4. 2013Describing Textures in the Wildcited by 2,110
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2019Natural Adversarial Examplescited by 1,401
  8. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  10. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  11. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  12. 2020In Search of Lost Domain Generalizationcited by 1,042
38 more
  1. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  2. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  3. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  4. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  5. 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
  6. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  7. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  8. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  9. 2022ViM: Out-Of-Distribution with Virtual-logit Matchingcited by 404
  10. 2018Recent Advances in Open Set Recognition: A Surveycited by 309
  11. 2022VOS: Learning What You Don't Know by Virtual Outlier Synthesiscited by 300
  12. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  13. 2022OpenOOD: Benchmarking Generalized Out-of-Distribution Detectioncited by 267
  14. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  15. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  16. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 178
  17. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  18. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  19. 2021OpenGAN: Open-Set Recognition via Open Data Generationcited by 135
  20. 2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
  21. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  22. 2020Hybrid Models for Open Set Recognitioncited by 109
  23. 2022POEM: Out-of-Distribution Detection with Posterior Samplingcited by 108
  24. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  25. 2020Conditional Gaussian Distribution Learning for Open Set Recognitioncited by 104
  26. 2018Generative Probabilistic Novelty Detection with Adversarial Autoencoderscited by 99
  27. 2023Vita-CLIP: Video and text adaptive CLIP via Multimodal Promptingcited by 90
  28. 2022Full-Spectrum Out-of-Distribution Detectioncited by 63
  29. 2023AD-CLIP: Adapting Domains in Prompt Space Using CLIPcited by 62
  30. 2023Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detectioncited by 57
  31. 2022Boosting Out-of-distribution Detection with Typical Featurescited by 48
  32. 2024Learning Transferable Negative Prompts for Out-of-Distribution Detectioncited by 43
  33. 2023ID-like Prompt Learning for Few-Shot Out-of-Distribution Detectioncited by 42
  34. 2014Maximin effects in inhomogeneous large-scale datacited by 32
  35. 2024Any-Shift Prompting for Generalization over Distributionscited by 24
  36. 2023Domain Aligned CLIP for Few-shot Classificationcited by 16
  37. 2023Understanding Hessian Alignment for Domain Generalizationcited by 15
  38. 2022Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answeringcited by 7

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Component-Based Out-of-Distribution Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Continual Test-Time Adaptation of Vision-Language Models in Open-Vocabulary Semantic Segmentation
  2. 2026TTL: Test-time Textual Learning for OOD Detection with Pretrained Vision-Language Models
  3. 2025An Empirical Analysis of VLM-based OOD Detection: Mechanisms, Advantages, and Sensitivity
  4. 2026OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models
  5. 2025Test-Time Consistency in Vision Language Models
  6. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Models
  7. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey
  8. 2026T-QPM: Enabling Temporal Out-Of-Distribution Detection and Domain Generalization for Vision-Language Models in Open-World