mlsanityData to 14 Oct 2026

BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models

arXiv 2312.02896 ↗First posted 5 Dec 2023cs.CVcited by 41

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Large Multimodal Models (LMMs) such as GPT-4V and LLaVA have shown remarkable capabilities in visual reasoning with common image styles. However, their robustness against diverse style shifts, crucial for practical applications, remains largely unexplored. In this paper, we propose a new benchmark, BenchLMM, to assess the robustness of LMMs against three different styles: artistic image style, imaging sensor style, and application style, where each style has five sub-styles. Utilizing BenchLMM, we comprehensively evaluate state-of-the-art LMMs and reveal: 1) LMMs generally suffer performance degradation when working with other styles; 2) An LMM performs better than another model in common style does not guarantee its superior performance in other styles; 3) LMMs' reasoning capability can be enhanced by prompting LMMs to predict the style first, based on which we propose a versatile and training-free method for improving LMMs; 4) An intelligent LMM is expected to interpret the causes of its errors when facing stylistic variations. We hope that our benchmark and analysis can shed new light on developing more intelligent and versatile LMMs.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 57 of its 69 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

57 of 69 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2023Chain of Thought Prompt Tuning in Vision Language Models16 Apr
  3. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
  4. 2023Logical Implications for Visual Question Answering Consistency16 Mar
  5. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb

Other references · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2015VQA: Visual Question Answeringcited by 2,506
38 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  2. 2019Towards VQA Models That Can Readcited by 1,718
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  7. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  12. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  13. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  14. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  15. 2017DOTA: A Large-scale Dataset for Object Detection in Aerial Imagescited by 623
  16. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  17. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  18. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  19. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  20. 2018nocaps: novel object captioning at scalecited by 444
  21. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  22. 2022Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detectioncited by 317
  23. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  24. 2021LLVIP: A Visible-infrared Paired Dataset for Low-light Visioncited by 231
  25. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  26. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 146
  27. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  28. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  29. 2016Dual Attention Networks for Multimodal Reasoning and Matchingcited by 114
  30. 2019Cycle-Consistency for Robust Visual Question Answeringcited by 105
  31. 2020Improving Image Captioning with Better Use of Captionscited by 84
  32. 2021Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answeringcited by 71
  33. 2019MUREL: Multimodal Relational Reasoning for Visual Question Answeringcited by 68
  34. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyondcited by 51
  35. 2022MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answeringcited by 36
  36. 2020Unshuffling Data for Improved Generalizationcited by 35
  37. 2021Greedy Gradient Ensemble for Robust Visual Question Answeringcited by 34
  38. 2023COCO-O: A Benchmark for Object Detectors under Natural Distribution Shiftscited by 34

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 32; 2025: 8; 2026: 1.

41 papers.

Sort:

By area:

  1. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  2. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  3. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  4. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 82
  5. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  6. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  7. 2024AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perceptioncited by 65
  8. 2024AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perceptioncited by 54
  9. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  10. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  11. 2024Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Modelscited by 47
  12. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
29 more
  1. 2024PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathologycited by 43
  2. 2024SHIELD : An Evaluation Benchmark for Face Spoofing and Forgery Detection with Multimodal Large Language Modelscited by 40
  3. 2024FakeBench: Probing Explainable Fake Image Detection via Large Multimodal Modelscited by 36
  4. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  5. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Modelscited by 32
  6. 2024CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoningcited by 31
  7. 2024How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMscited by 30
  8. 2024DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layercited by 27
  9. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  10. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  11. 2024Quality Assessment in the Era of Large Models: A Surveycited by 26
  12. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 25
  13. 2024Benchmarking Large Multimodal Models against Common Corruptionscited by 24
  14. 2025Geolocation with Real Human Gameplay Data: A Large-Scale Dataset and Human-Like Reasoning Frameworkcited by 24
  15. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 21
  16. 20242AFC Prompting of Large Multimodal Models for Image Quality Assessmentcited by 20
  17. 2024Hazards in Daily Life? Enabling Robots to Proactively Detect and Resolve Anomaliescited by 19
  18. 2025RvLLM: LLM Runtime Verification with Domain Knowledgecited by 17
  19. 2024II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Modelscited by 16
  20. 2024Towards Data-Centric Face Anti-Spoofing: Improving Cross-domain Generalization via Physics-based Data Synthesiscited by 16
  21. 2024Can MLLMs Understand the Deep Implication Behind Chinese Images?cited by 14
  22. 2025Towards Cross-View Point Correspondence in Vision-Language Modelscited by 14
  23. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learningcited by 11
  24. 2024ActiView: Evaluating Active Perception Ability for Multimodal Large Language Modelscited by 11
  25. 2024MMAC-Copilot: Multi-modal Agent Collaboration Operating Copilotcited by 8
  26. 2024A Survey of Calibration Process for Black-Box LLMscited by 8
  27. 2025Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigationcited by 4
  28. 2025From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationshipcited by 2
  29. 2026Knowledge Visualization: A Benchmark and Method for Knowledge-Intensive Text-to-Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models
  2. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
  3. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
  4. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorial
  5. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
  6. 2026Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?
  7. 2024R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling