# Envisioning Outlier Exposure by Large Language Models for Out-of-Distribution Detection

arXiv: https://arxiv.org/abs/2406.00806 · HTML: https://mlsanity.com/paper/2406.00806/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 62 references that are in the map.
- First posted on arXiv 2 Jun 2024.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)

## Built on by (most cited first, 30 of 32)

- [Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey](https://mlsanity.com/paper/2407.21794/) (2024)
- [Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advances](https://mlsanity.com/paper/2409.11884/) (2024)
- [From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?](https://mlsanity.com/paper/2506.08295/) (2025)
- [FOOGD: Federated Collaboration for Both Out-of-distribution Generalization and Detection](https://mlsanity.com/paper/2410.11397/) (2024)
- [Noisy Test-Time Adaptation in Vision-Language Models](https://mlsanity.com/paper/2502.14604/) (2025)
- [ANTS: Adaptive Negative Textual Space Shaping for OOD Detection via Test-Time MLLM Understanding and Reasoning](https://mlsanity.com/paper/2509.03951/) (2025)
- [Adaptive Multi-prompt Contrastive Network for Few-shot Out-of-distribution Detection](https://mlsanity.com/paper/2506.17633/) (2025)
- [Mysteries of the Deep: Role of Intermediate Representations in Out of Distribution Detection](https://mlsanity.com/paper/2510.05782/) (2025)
- [Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMs](https://mlsanity.com/paper/2510.20064/) (2025)
- [Graph Synthetic Out-of-Distribution Exposure with Large Language Models](https://mlsanity.com/paper/2504.21198/) (2025)
- [Recent Advances in Out-of-Distribution Detection with CLIP-Like Models: A Survey](https://mlsanity.com/paper/2505.02448/) (2025)
- [Reference-guided Policy Optimization for Molecular Optimization via LLM Reasoning](https://mlsanity.com/paper/2603.05900/) (2026)
- [Revisiting Logit Distributions for Reliable Out-of-Distribution Detection](https://mlsanity.com/paper/2510.20134/) (2025)
- [Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs](https://mlsanity.com/paper/2605.21602/) (2026)
- [A Benchmark and Evaluation for Real-World Out-of-Distribution Detection Using Vision-Language Models](https://mlsanity.com/paper/2501.18463/) (2025)
- [Mitigating the Modality Gap: Few-Shot Out-of-Distribution Detection with Multi-modal Prototypes and Image Bias Estimation](https://mlsanity.com/paper/2502.00662/) (2025)
- [Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracy](https://mlsanity.com/paper/2509.21173/) (2025)
- [Reflexive Guidance: Improving OoDD in Vision-Language Models via Self-Guided Image-Adaptive Concept Generation](https://mlsanity.com/paper/2410.14975/) (2024)
- [GLIP-OOD: Zero-Shot Graph OOD Detection with Graph Foundation Model](https://mlsanity.com/paper/2504.21186/) (2025)
- [Activation Matters: Test-time Activated Negative Labels for OOD Detection with Vision-Language Models](https://mlsanity.com/paper/2603.25250/) (2026)
- [OT-DETECTOR: Delving into Optimal Transport for Zero-shot Out-of-Distribution Detection](https://mlsanity.com/paper/2503.06442/) (2025)
- [Mind the Way You Select Negative Texts: Pursuing the Distance Consistency in OOD Detection with VLMs](https://mlsanity.com/paper/2603.02618/) (2026)
- [Vision Also You Need: Navigating Out-of-Distribution Detection with Multimodal Large Language Model](https://mlsanity.com/paper/2601.14052/) (2026)
- [Component-Based Out-of-Distribution Detection](https://mlsanity.com/paper/2604.21546/) (2026)
- [TINS: Test-time ID-prototype-separated Negative Semantics Learning for OOD Detection](https://mlsanity.com/paper/2605.10756/) (2026)
- [NegAS: Negative Label Guided Attention and Scoring for Out-of-Distribution Object Detection with Vision-Language Models](https://mlsanity.com/paper/2606.22537/) (2026)
- [The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Model](https://mlsanity.com/paper/2607.13660/) (2026)
- [LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation](https://mlsanity.com/paper/2607.17653/) (2026)
- [A Taxonomy of Cognitive Capability Gaps in Generative and Agentic AI](https://mlsanity.com/paper/2608.02553/) (2026)
- [Fi-ImageNet-1k: An OOD Benchmark From the Inside of the ImageNet-1k Validation Set](https://mlsanity.com/paper/2609.01027/) (2026)

## Abstract (arXiv)

Detecting out-of-distribution (OOD) samples is essential when deploying machine learning models in open-world scenarios. Zero-shot OOD detection, requiring no training on in-distribution (ID) data, has been possible with the advent of vision-language models like CLIP. Existing methods build a text-based classifier with only closed-set labels. However, this largely restricts the inherent capability of CLIP to recognize samples from large and open label space. In this paper, we propose to tackle this constraint by leveraging the expert knowledge and reasoning capability of large language models (LLM) to Envision potential Outlier Exposure, termed EOE, without access to any actual OOD data. Owing to better adaptation to open-world scenarios, EOE can be generalized to different tasks, including far, near, and fine-grained OOD detection. Technically, we design (1) LLM prompts based on visual similarity to generate potential outlier class labels specialized for OOD detection, as well as (2) a new score function based on potential outlier penalty to distinguish hard OOD samples effectively. Empirically, EOE achieves state-of-the-art performance across different OOD tasks and can be effectively scaled to the ImageNet-1K dataset. The code is publicly available at: https://github.com/tmlr-group/EOE.