mlsanityData to 7 Oct 2026

Benchmarking Large Multimodal Models against Common Corruptions

arXiv 2401.11943 ↗First posted 22 Jan 2024cs.LG cs.CL cs.CRcited by 22

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

This technical report aims to fill a deficiency in the assessment of large multimodal models (LMMs) by specifically examining the self-consistency of their outputs when subjected to common corruptions. We investigate the cross-modal interactions between text, image, and speech, encompassing four essential generation tasks: text-to-image, image-to-text, text-to-speech, and speech-to-text. We create a comprehensive benchmark, named MMCBench, that covers more than 100 popular LMMs (totally over 150 model checkpoints). A thorough evaluation under common corruptions is critical for practical deployment and facilitates a better understanding of the reliability of cutting-edge LMMs. The benchmarking code is available at https://github.com/sail-sg/MMCBench

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 128 references that are in the map. First posted on arXiv 22 Jan 2024.

Builds on

52 of 128 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models5 Dec
  2. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs27 Nov
  3. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model28 Apr

Other references · 49

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
37 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  5. 2023Adversarial Diffusion Distillationcited by 741
  6. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  7. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  8. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  9. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  10. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  11. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  12. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  13. 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 332
  14. 2023Are aligned neural networks adversarially aligned?cited by 305
  15. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  16. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  17. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulecited by 211
  18. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  19. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  20. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  21. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  22. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  23. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  24. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  25. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  26. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 137
  27. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  28. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  29. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 124
  30. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 120
  31. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  32. 2019Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systemscited by 99
  33. 2023Towards Efficient Visual Adaption via Structural Re-parameterizationcited by 73
  34. 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 72
  35. 2020TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLPcited by 45
  36. 2022Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shiftcited by 24
  37. 2023BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusioncited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 6; 2026: 8.

22 papers.

Sort:

By area:

  1. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  2. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  3. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  4. 2024R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?cited by 29
  5. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  6. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  7. 2024B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructionscited by 26
  8. 2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
  9. 2024LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Modelscited by 7
  10. 2025REVAL: A Comprehension Evaluation on Reliability and Values of Large Vision-Language Modelscited by 5
  11. 2024Evaluating Durability: Benchmark Insights into Multimodal Watermarkingcited by 3
  12. 2025From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationshipcited by 2
10 more
  1. 2026Are Video Reasoning Models Ready to Go Outside?cited by 2
  2. 2026DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verificationcited by 2
  3. 2025Model Hemorrhage and the Robustness Limits of Large Language Modelscited by 1
  4. 2025LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmologycited by 1
  5. 2026Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Modelscited by 1
  6. 2026Confidence-Aware Tool Orchestration for Robust Video Understandingcited by 1
  7. 2026Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoningcited by 1
  8. 2026Robust Test-time Video-Text Retrieval: Benchmarking and Adapting for Query Shiftscited by 0
  9. 2026How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbationscited by 0
  10. 2026Looks the Same, Answers Differently: Flip-Direction Steering for Robust Vision-Language Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
  2. 2024MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspective
  3. 2024Cross-Modal Consistency in Multimodal Large Language Models
  4. 2023Multimodal Large Language Models: A Survey
  5. 2025LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs
  6. 2026What We are Missing in Multimodal LLM Evaluation?
  7. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  8. 2025Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs