mlsanityData to 14 Oct 2026

CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models

arXiv 2402.13607 ↗First posted 21 Feb 2024cs.CV cs.CLcited by 11

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) have demonstrated promising results in a variety of tasks that combine vision and language. As these models become more integral to research and applications, conducting comprehensive evaluations of their capabilities has grown increasingly important. However, most existing benchmarks fail to consider that, in certain situations, images need to be interpreted within a broader context. In this work, we introduce a new benchmark, named as CODIS, designed to assess the ability of models to use context provided in free-form text to enhance visual comprehension. Our findings indicate that MLLMs consistently fall short of human performance on this benchmark. Further analysis confirms that these models struggle to effectively extract and utilize contextual information to improve their understanding of images. This underscores the pressing need to enhance the ability of MLLMs to comprehend visuals in a context-dependent manner. View our project website at https://thunlp-mt.github.io/CODIS.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 48 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

26 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Generative Multimodal Models are In-Context Learners20 Dec
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  3. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep

Other references · 21

  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  4. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  5. 2019Object Detection in 20 Years: A Surveycited by 531
  6. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  7. 2016Visual Dialogcited by 377
  8. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  9. 2018Graph R-CNN for Scene Graph Generationcited by 298
  10. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  11. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  12. 2022Scene Graph Generation: A Comprehensive Surveycited by 73
9 more
  1. 2022MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversationcited by 56
  2. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 51
  3. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertisecited by 49
  4. 2022Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimationcited by 43
  5. 2022Semantic-Conditional Diffusion Networks for Image Captioningcited by 38
  6. 2020Audio-Visual Floorplan Reconstructioncited by 32
  7. 2023Context Understanding in Computer Vision: A Surveycited by 23
  8. 2021Multi-label Iterated Learning for Image Classification with Label Ambiguitycited by 19
  9. 2022Ambiguous Images With Human Judgments for Robust Visual Event Classificationcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 3; 2026: 3.

11 papers.

  1. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 104
  2. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  3. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  4. 2024Large Language Model Benchmarks in Medical Taskscited by 17
  5. 2024ActiView: Evaluating Active Perception Ability for Multimodal Large Language Modelscited by 11
  6. 2024Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusionscited by 11
  7. 2025CoSpace: Benchmarking Continuous Space Perception Ability for Vision-Language Modelscited by 2
  8. 2025SAP-Bench: Benchmarking Multimodal Large Language Models in Surgical Action Planningcited by 1
  9. 2026Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Actcited by 1
  10. 2026ROSE: Benchmarking the Perception-to-Action Gap in Multimodal Modelscited by 0
  11. 2026SAVOR: Self-Aware Visual Grounding via Confidence-Calibrated Reinforcement Learning for Multimodal Hallucination Mitigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension
  2. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
  3. 2024A Survey on Benchmarks of Multimodal Large Language Models
  4. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
  5. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
  6. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  7. 2024Can MLLMs Perform Text-to-Image In-Context Learning?
  8. 2024MileBench: Benchmarking MLLMs in Long Context