mlsanityData to 7 Oct 2026

From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities

arXiv 2401.15071 ↗First posted 26 Jan 2024cs.CVcited by 18

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multi-modal Large Language Models (MLLMs) have shown impressive abilities in generating reasonable responses with respect to multi-modal contents. However, there is still a wide gap between the performance of recent MLLM-based applications and the expectation of the broad public, even though the most powerful OpenAI's GPT-4 and Google's Gemini have been deployed. This paper strives to enhance understanding of the gap through the lens of a qualitative study on the generalizability, trustworthiness, and causal reasoning capabilities of recent proprietary and open-source MLLMs across four modalities: ie, text, code, image, and video, ultimately aiming to improve the transparency of MLLMs. We believe these properties are several representative factors that define the reliability of MLLMs, in supporting various downstream applications. To be specific, we evaluate the closed-source GPT-4 and Gemini and 6 open-source LLMs and MLLMs. Overall we evaluate 230 manually designed cases, where the qualitative results are then summarized into 12 scores (ie, 4 modalities times 3 properties). In total, we uncover 14 empirical findings that are useful to understand the capabilities and limitations of both proprietary and open-source MLLMs, towards more reliable downstream multi-modal applications.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 74 references that are in the map. First posted on arXiv 26 Jan 2024.

Builds on

39 of 74 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models5 Nov
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep

Other references · 37

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2024Mixtral of Expertscited by 1,785
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
25 more
  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2021Measuring Coding Challenge Competence With APPScited by 879
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  4. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  5. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  6. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  7. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  8. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  9. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 262
  10. 2023AI Alignment: A Comprehensive Surveycited by 256
  11. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  12. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  13. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  14. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  15. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  16. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  17. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  18. 2023Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluationcited by 69
  19. 2023Comparing Code Explanations Created by Students and Large Language Modelscited by 63
  20. 2023Fake Alignment: Are LLMs Really Aligned Well?cited by 39
  21. 2023Flames: Benchmarking Value Alignment of LLMs in Chinesecited by 35
  22. 2021CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Modelscited by 33
  23. 2023LLMEval: A Preliminary Study on How to Evaluate Large Language Modelscited by 23
  24. 2023Controlling risks of AI in chemical science with agentscited by 17
  25. 2023Privacy-Aware Document Visual Question Answeringcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 7; 2026: 1.

18 papers.

Sort:

By area:

  1. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  2. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  3. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
  4. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  5. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  6. 2025Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injectioncited by 23
  7. 2024Causal Evaluation of Language Modelscited by 18
  8. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answeringcited by 10
  9. 2025Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Modelscited by 10
  10. 2025CLINIC: Evaluating Multilingual Trustworthiness in Language Models for Healthcarecited by 7
  11. 2024Principled Understanding of Generalization for Generative Transformer Models in Arithmetic Reasoning Taskscited by 5
  12. 2024The Tug of War Within: Mitigating the Fairness-Privacy Conflicts in Large Language Modelscited by 5
6 more
  1. 2025Dynamic Chain-of-Thought: Towards Adaptive Deep Reasoningcited by 5
  2. 2024A Misleading Gallery of Fluid Motion by Generative Artificial Intelligencecited by 3
  3. 2025Contextual Image Attack: How Visual Context Exposes Multimodal Safety Vulnerabilitiescited by 2
  4. 2025Towards Aligning Multimodal LLMs with Human Experts: A Focus on Parent-Child Interactioncited by 1
  5. 2026All Changes May Have Invariant Principles: Improving Ever-Shifting Harmful Meme Detection via Design Concept Reproductioncited by 1
  6. 2025Wormhole Memory: A Rubik's Cube for Cross-Dialogue Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Models
  2. 2025Benchmarking Large and Small MLLMs
  3. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  4. 2025On Path to Multimodal Generalist: General-Level and General-Bench
  5. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveys
  6. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
  7. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective
  8. 2025MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI