mlsanityData to 7 Oct 2026

Difficult Task Yes but Simple Task No: Unveiling the Laziness in Multimodal LLMs

arXiv 2410.11437 ↗First posted 15 Oct 2024cs.CL cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) demonstrate a strong understanding of the real world and can even handle complex tasks. However, they still fail on some straightforward visual question-answering (VQA) problems. This paper dives deeper into this issue, revealing that models tend to err when answering easy questions (e.g. Yes/No questions) about an image, even though they can correctly describe it. We refer to this model behavior discrepancy between difficult and simple questions as model laziness. To systematically investigate model laziness, we manually construct LazyBench, a benchmark that includes Yes/No, multiple choice, short answer questions, and image description tasks that are related to the same subjects in the images. Based on LazyBench, we observe that laziness widely exists in current advanced MLLMs (e.g. GPT-4o, Gemini-1.5-pro, Claude 3 and LLaVA-v1.5-13B), and it is more pronounced on stronger models. We also analyze the VQA v2 (LLaVA-v1.5-13B) benchmark and find that about half of its failure cases are caused by model laziness, which further highlights the importance of ensuring that the model fully utilizes its capability. To this end, we conduct preliminary exploration on how to mitigate laziness and find that chain of thought (CoT) can effectively address this issue.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 38 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

21 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  11. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  12. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
7 more
  1. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  2. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  3. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuningcited by 69
  4. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67
  5. 2022There is a Time and Place for Reasoning Beyond the Imagecited by 22
  6. 2023Generate then Select: Open-ended Visual Question Answering Guided by World Knowledgecited by 16
  7. 2024A Better LLM Evaluator for Text Generation: The Impact of Prompt Output Sequencing and Optimizationcited by 11

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  2. 2025Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessmentcited by 7
  3. 2025Do "New Snow Tablets" Contain Snow? Large Language Models Over-Rely on Names to Identify Ingredients of Chinese Drugscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation
  2. 2025SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models
  3. 2025Evaluating Variance in Visual Question Answering Benchmarks
  4. 2024Efficient Multimodal Large Language Models: A Survey
  5. 2024Revisiting Multi-Modal LLM Evaluation
  6. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly
  7. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  8. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models