mlsanityData to 7 Oct 2026

MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Models

arXiv 2607.27637 ↗First posted 30 Jul 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have achieved strong performance on a wide range of vision-language tasks, but often fail under imperfect or shifted contexts. A reliable MLLM should refuse truly out-of-context (OOC) questions with subject-level context shifts while still answering shifted in-context (Shifted IC) questions with non-subject context shifts. Existing benchmarks mainly target OOC or visually unanswerable questions, but overlook answerable Shifted IC cases and cover limited OOC shifts. To fill this gap, we present MMOOC, a large-scale benchmark for evaluating refusal and robust answering abilities of MLLMs. MMOOC contains over 41K image-question pairs, including answerable Shifted IC cases and unanswerable OOC cases, spanning three question formats, eight shift types and six visual scenarios, with data quality ensured through MLLM-based filtering and human verification. We evaluate model responses using Accuracy and Refusal Rate, and further introduce an LLM-as-a-Judge metric to assess the correctness of model reasoning. Experiments on diverse MLLMs show that current models still struggle to balance answer-ability and refusal under shifted contexts. We further analyze key failure patterns and show that post-training can improve robustness. MMOOC will be made publicly available.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 61 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

50 of 61 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Qwen3-VL Technical Report26 Nov
  2. 2025MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questions29 Jul
  3. 2024MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Models17 Jun
  4. 2024Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Models29 Mar
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 43

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2024GPT-4o System Cardcited by 4,467
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
31 more
  1. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  2. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  3. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  4. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  5. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  6. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 85
  7. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  8. 2024SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detectioncited by 73
  9. 2023Alignment for Honestycited by 72
  10. 2023FELM: Benchmarking Factuality Evaluation of Large Language Modelscited by 66
  11. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Modelscited by 51
  12. 2024HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoningcited by 30
  13. 2024LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Modelscited by 21
  14. 2024A Survey on the Honesty of Large Language Modelscited by 21
  15. 2024Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulnesscited by 17
  16. 2024HonestLLM: Toward an Honest and Helpful Large Language Modelcited by 17
  17. 2025MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLMcited by 17
  18. 2026ConeSep: Cone-based Robust Noise-Unlearning Compositional Network for Composed Image Retrievalcited by 15
  19. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledgecited by 14
  20. 2025ANTS: Adaptive Negative Textual Space Shaping for OOD Detection via Test-Time MLLM Understanding and Reasoningcited by 10
  21. 2026COMBINER: Composed Image Retrieval Guided by Attribute-based Neighbor Relationscited by 9
  22. 2025RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Modelscited by 8
  23. 2025Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMscited by 8
  24. 2026SPPO: Sequence-Level PPO for Long-Horizon Reasoning Taskscited by 7
  25. 2025CLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answeringcited by 6
  26. 2025Hybrid Meta-learners for Estimating Heterogeneous Treatment Effectscited by 4
  27. 2025Knowledge Regularized Negative Feature Tuning of Vision-Language Models for Out-of-Distribution Detectioncited by 4
  28. 2026Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decodingcited by 3
  29. 2026OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Predictioncited by 2
  30. 2026Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMscited by 1
  31. 2026Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation
  2. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  3. 2025Evaluating Variance in Visual Question Answering Benchmarks
  4. 2024A Survey on Benchmarks of Multimodal Large Language Models
  5. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation
  6. 2026MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models
  7. 2026MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems
  8. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models