mlsanityData to 7 Oct 2026

Can Large Multimodal Models Uncover Deep Semantics Behind Images?

arXiv 2402.11281 ↗First posted 17 Feb 2024cs.CLcited by 16

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Understanding the deep semantics of images is essential in the era dominated by social media. However, current research works primarily on the superficial description of images, revealing a notable deficiency in the systematic investigation of the inherent deep semantics. In this work, we introduce DEEPEVAL, a comprehensive benchmark to assess Large Multimodal Models' (LMMs) capacities of visual deep semantics. DEEPEVAL includes human-annotated dataset and three progressive subtasks: fine-grained description selection, in-depth title matching, and deep semantics understanding. Utilizing DEEPEVAL, we evaluate 9 open-source LMMs and GPT-4V(ision). Our evaluation demonstrates a substantial gap between the deep semantic comprehension capabilities of existing LMMs and humans. For example, GPT-4V is 30% behind humans in understanding deep semantics, even though it achieves human-comparable performance in image description. Further analysis reveals that LMM performance on DEEPEVAL varies according to the specific facets of deep semantics explored, indicating the fundamental challenges remaining in developing LMMs.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 75 references that are in the map. First posted on arXiv 17 Feb 2024.

Builds on

30 of 75 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks13 Oct
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2020GLU Variants Improve Transformercited by 1,724
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
13 more
  1. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  2. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  3. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  4. 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 173
  5. 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 110
  6. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  7. 2022Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancementcited by 26
  8. 2021Nice perfume. How long did you marinate in it? Multimodal Sarcasm Explanationcited by 25
  9. 2023Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generationcited by 19
  10. 2015Humor in Collective Discourse: Unsupervised Funniness Detection in the New Yorker Cartoon Caption Contestcited by 16
  11. 2023The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisationcited by 12
  12. 2022Co-VQA : Answering by Interactive Sub Question Sequencecited by 8
  13. 2023ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categoriescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 4.

16 papers.

Sort:

By area:

  1. 2025MemInsight: Autonomous Memory Augmentation for LLM Agentscited by 84
  2. 2024Can MLLMs Understand the Deep Implication Behind Chinese Images?cited by 14
  3. 2024Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictionscited by 11
  4. 2024MVP-Bench: Can Large Vision--Language Models Conduct Multi-level Visual Perception Like Humans?cited by 8
  5. 2024PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehensioncited by 5
  6. 2025When 'YES' Meets 'BUT': Can Large Models Comprehend Contradictory Humor Through Comparative Reasoning?cited by 5
  7. 2025Let Androids Dream of Electric Sheep: A Human-Inspired Image Implication Understanding and Reasoning Frameworkcited by 5
  8. 2025Beyond Single Frames: Can LMMs Comprehend Temporal and Contextual Narratives in Image Sequences?cited by 4
  9. 2025InsightVision: A Comprehensive, Multi-Level Chinese-based Benchmark for Evaluating Implicit Visual Semantics in Large Vision Language Modelscited by 4
  10. 2026The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agentscited by 4
  11. 2025Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understandingcited by 3
  12. 2025Seeing is Not Understanding: A Benchmark on Perception-Cognition Disparities in Large Language Modelscited by 0
4 more
  1. 2025Visual Room 2.0: Seeing is Not Understanding for MLLMscited by 0
  2. 2026Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understandingcited by 0
  3. 2026Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challengescited by 0
  4. 2026Towards High-Level Semantic Intelligencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  2. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
  3. 2024Probing Multimodal Large Language Models for Global and Local Semantic Representations
  4. 2026Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models
  5. 2025On Large Multimodal Models as Open-World Image Classifiers
  6. 2026Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Models
  7. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models
  8. 2024Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion