mlsanityData to 7 Oct 2026

Exploring Large Language Models for Multi-Modal Out-of-Distribution Detection

arXiv 2310.08027 ↗First posted 12 Oct 2023cs.CL cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Out-of-distribution (OOD) detection is essential for reliable and trustworthy machine learning. Recent multi-modal OOD detection leverages textual information from in-distribution (ID) class names for visual OOD detection, yet it currently neglects the rich contextual information of ID classes. Large language models (LLMs) encode a wealth of world knowledge and can be prompted to generate descriptive features for each class. Indiscriminately using such knowledge causes catastrophic damage to OOD detection due to LLMs' hallucinations, as is observed by our analysis. In this paper, we propose to apply world knowledge to enhance OOD detection performance through selective generation from LLMs. Specifically, we introduce a consistency-based uncertainty calibration method to estimate the confidence score of each generation. We further extract visual objects from each image to fully capitalize on the aforementioned world knowledge. Extensive experiments demonstrate that our method consistently outperforms the state-of-the-art.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 46 of its 64 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

46 of 64 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  11. 2016Concrete Problems in AI Safetycited by 1,582
  12. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
34 more
  1. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  2. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  3. 2023Teaching Large Language Models to Self-Debugcited by 963
  4. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  5. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  6. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  7. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  8. 2020Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Datacited by 408
  9. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  10. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  11. 2022Visual Classification via Description from Large Language Modelscited by 331
  12. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  13. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  14. 2022DETRs with Collaborative Hybrid Assignments Trainingcited by 273
  15. 2022Prompting GPT-3 To Be Reliablecited by 258
  16. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  17. 2014Towards Open World Recognitioncited by 226
  18. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  19. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200
  20. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 193
  21. 2020Contrastive Training for Improved Out-of-Distribution Detectioncited by 186
  22. 2021Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIPcited by 151
  23. 2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
  24. 2021Contrastive Out-of-Distribution Detection for Pretrained Transformerscited by 70
  25. 2021On the Impact of Spurious Correlation for Out-of-distribution Detectioncited by 68
  26. 2023YOLOv6 v3.0: A Full-Scale Reloadingcited by 58
  27. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 49
  28. 2021Out-of-Scope Intent Detection with Self-Supervision and Discriminative Trainingcited by 42
  29. 2022BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-trainingcited by 33
  30. 2023A Survey on Out-of-Distribution Detection in NLPcited by 31
  31. 2021Enhancing the Generalization for Intent Classification and Out-of-Domain Detection in SLUcited by 22
  32. 2022Estimating Soft Labels for Out-of-Domain Intent Detectioncited by 11
  33. 2023Long-Tailed Question Answering in an Open Worldcited by 7
  34. 2023Domain Incremental Lifelong Learning in an Open Worldcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 6; 2026: 2.

12 papers.

  1. 2023Open-Set Image Tagging with Multi-Grained Text Supervisioncited by 67
  2. 2024Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advancescited by 24
  3. 2025ANTS: Adaptive Negative Textual Space Shaping for OOD Detection via Test-Time MLLM Understanding and Reasoningcited by 10
  4. 2024SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximationcited by 9
  5. 2025SimLabel: Consistency-Guided OOD Detection with Pretrained Vision-Language Modelscited by 5
  6. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 3
  7. 2025Spurious-Aware Prototype Refinement for Reliable Out-of-Distribution Detectioncited by 2
  8. 2025Analysis of Image-and-Text Uncertainty Propagation in Multimodal Large Language Models with Cardiac MR-Based Applicationscited by 1
  9. 2024'No' Matters: Out-of-Distribution Detection in Multimodality Long Dialoguecited by 0
  10. 2025STEMTOX: From Collaborative Tags to Fine-Grained Toxic Meme Detection via Entropy-Guided Multi-Task Learningcited by 0
  11. 2026Multi-Agent Cooperative Learning for Robust Vision-Language Alignment under OOD Conceptscited by 0
  12. 2026Bridging the Semantic Chasm: Synergistic Conceptual Anchoring for Generalized Few-Shot and Zero-Shot OOD Perceptioncited by 0