# MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities

arXiv: https://arxiv.org/abs/2405.17419 · HTML: https://mlsanity.com/paper/2405.17419/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 73 references that are in the map.
- First posted on arXiv 27 May 2024.

## Builds on (most cited first, 30 of 54)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [ReAct: Out-of-distribution Detection With Rectified Activations](https://mlsanity.com/paper/2111.12797/) (2021)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [ViM: Out-Of-Distribution with Virtual-logit Matching](https://mlsanity.com/paper/2203.10807/) (2022)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [VOS: Learning What You Don't Know by Virtual Outlier Synthesis](https://mlsanity.com/paper/2202.01197/) (2022)
- [Deep Semi-Supervised Anomaly Detection](https://mlsanity.com/paper/1906.02694/) (2019)
- [OpenOOD: Benchmarking Generalized Out-of-Distribution Detection](https://mlsanity.com/paper/2210.07242/) (2022)

## Built on by (most cited first, 28 of 28)

- [Treble Counterfactual VLMs: A Causal Approach to Hallucination](https://mlsanity.com/paper/2503.06169/) (2025)
- [Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advances](https://mlsanity.com/paper/2409.11884/) (2024)
- [AStar: Boosting Multimodal Reasoning with Automated Structured Thinking](https://mlsanity.com/paper/2502.02339/) (2025)
- [DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learning](https://mlsanity.com/paper/2503.11892/) (2025)
- [Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models](https://mlsanity.com/paper/2501.18592/) (2025)
- [Secure On-Device Video OOD Detection Without Backpropagation](https://mlsanity.com/paper/2503.06166/) (2025)
- [Towards Multimodal Open-Set Domain Generalization and Adaptation through Self-supervision](https://mlsanity.com/paper/2407.01518/) (2024)
- [MetaOOD: Automatic Selection of OOD Detection Models](https://mlsanity.com/paper/2410.03074/) (2024)
- [Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization](https://mlsanity.com/paper/2501.13924/) (2025)
- [LEGO-Learn: Label-Efficient Graph Open-Set Learning](https://mlsanity.com/paper/2410.16386/) (2024)
- [ClimateLLM: Efficient Weather Forecasting via Frequency-Aware Large Language Models](https://mlsanity.com/paper/2502.11059/) (2025)
- [When Continue Learning Meets Multimodal Large Language Model: A Survey](https://mlsanity.com/paper/2503.01887/) (2025)
- [GEM: Gaussian Embedding Modeling for Out-of-Distribution Detection in GUI Agents](https://mlsanity.com/paper/2505.12842/) (2025)
- [Extremely Simple Multimodal Outlier Synthesis for Out-of-Distribution Detection and Segmentation](https://mlsanity.com/paper/2505.16985/) (2025)
- [Recall and Refine: A Simple but Effective Source-free Open-set Domain Adaptation Framework](https://mlsanity.com/paper/2411.12558/) (2024)
- [Graph Synthetic Out-of-Distribution Exposure with Large Language Models](https://mlsanity.com/paper/2504.21198/) (2025)
- [Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradation](https://mlsanity.com/paper/2506.17442/) (2025)
- [FilterRAG: Zero-Shot Informed Retrieval-Augmented Generation to Mitigate Hallucinations in VQA](https://mlsanity.com/paper/2502.18536/) (2025)
- [To Trust Or Not To Trust Your Vision-Language Model's Prediction](https://mlsanity.com/paper/2505.23745/) (2025)
- [Relative Energy Learning for LiDAR Out-of-Distribution Detection](https://mlsanity.com/paper/2511.06720/) (2025)
- [BelHouse3D: A Benchmark Dataset for Assessing Occlusion Robustness in 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2411.13251/) (2024)
- [Safe Uncertainty-Aware Learning of Robotic Suturing](https://mlsanity.com/paper/2505.16596/) (2025)
- [RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer](https://mlsanity.com/paper/2506.11465/) (2025)
- [Adaptive Confidence Regularization for Multimodal Failure Detection](https://mlsanity.com/paper/2603.02200/) (2026)
- [Multimodal Learning for Arcing Detection in Pantograph-Catenary Systems](https://mlsanity.com/paper/2602.08792/) (2026)
- [Modality-Aware Out-of-Distribution Detection for Multi-Modal Action Recognition](https://mlsanity.com/paper/2606.24404/) (2026)
- [Additive Causal Construction for Transferable and Reconfigurable Cross-System Learning in Multi-Source Image Fusion](https://mlsanity.com/paper/2607.02572/) (2026)
- [Training-Free Out-of-Distribution Detection for Pathology Whole-Slide Images](https://mlsanity.com/paper/2608.01407/) (2026)

## Abstract (arXiv)

Detecting out-of-distribution (OOD) samples is important for deploying machine learning models in safety-critical applications such as autonomous driving and robot-assisted surgery. Existing research has mainly focused on unimodal scenarios on image data. However, real-world applications are inherently multimodal, which makes it essential to leverage information from multiple modalities to enhance the efficacy of OOD detection. To establish a foundation for more realistic Multimodal OOD Detection, we introduce the first-of-its-kind benchmark, MultiOOD, characterized by diverse dataset sizes and varying modality combinations. We first evaluate existing unimodal OOD detection algorithms on MultiOOD, observing that the mere inclusion of additional modalities yields substantial improvements. This underscores the importance of utilizing multiple modalities for OOD detection. Based on the observation of Modality Prediction Discrepancy between in-distribution (ID) and OOD data, and its strong correlation with OOD performance, we propose the Agree-to-Disagree (A2D) algorithm to encourage such discrepancy during training. Moreover, we introduce a novel outlier synthesis method, NP-Mix, which explores broader feature spaces by leveraging the information from nearest neighbor classes and complements A2D to strengthen OOD detection performance. Extensive experiments on MultiOOD demonstrate that training with A2D and NP-Mix improves existing OOD detection algorithms by a large margin. Our source code and MultiOOD benchmark are available at https://github.com/donghao51/MultiOOD.