mlsanityData to 7 Oct 2026

MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?

arXiv 2408.13257 ↗First posted 23 Aug 2024cs.CVcited by 284

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Comprehensive evaluation of Multimodal Large Language Models (MLLMs) has recently garnered widespread attention in the research community. However, we observe that existing benchmarks present several common barriers that make it difficult to measure the significant challenges that models face in the real world, including: 1) small data scale leads to a large performance variance; 2) reliance on model-based annotations results in restricted data quality; 3) insufficient task difficulty, especially caused by the limited image resolution. To tackle these issues, we introduce MME-RealWorld. Specifically, we collect more than 300K images from public datasets and the Internet, filtering 13,366 high-quality images for annotation. This involves the efforts of professional 25 annotators and 7 experts in MLLMs, contributing to 29,429 question-answer pairs that cover 43 subtasks across 5 real-world scenarios, extremely challenging even for humans. As far as we know, MME-RealWorld is the largest manually annotated benchmark to date, featuring the highest resolution and a targeted focus on real-world applications. We further conduct a thorough evaluation involving 28 prominent MLLMs, such as GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet. Our results show that even the most advanced models struggle with our benchmarks, where none of them reach 60% accuracy. The challenges of perceiving high-resolution images and understanding complex real-world scenarios remain urgent issues to be addressed. The data and evaluation code are released at https://mme-realworld.github.io/ .

Key facts. Cited by 284 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 72 references that are in the map. First posted on arXiv 23 Aug 2024.

Builds on

62 of 72 references.

Closest ancestors · 21

Same direction, newest first

  1. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLM9 Aug
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  3. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension25 Apr
  4. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  5. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  6. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  8. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models28 Nov
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  10. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  11. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  12. 2023TouchStone: Evaluating Vision-Language Models by Language Models31 Aug
9 more
  1. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  5. 2023A Survey on Multimodal Large Language Models23 Jun
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  7. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  8. 2023Mistral 7Bcited by 3,065
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
29 more
  1. 2023PaLM 2 Technical Reportcited by 1,154
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  4. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  6. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  7. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  8. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  9. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  10. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  11. 2020Detection and Tracking Meet Drones Challengecited by 261
  12. 2021LLVIP: A Visible-infrared Paired Dataset for Low-light Visioncited by 229
  13. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  14. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  15. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  16. 2021FAIR1M: A Benchmark Dataset for Fine-grained Object Recognition in High-Resolution Remote Sensing Imagerycited by 143
  17. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  18. 2022CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Drivingcited by 100
  19. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  20. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  21. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 75
  22. 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63
  23. 2024Embodied Understanding of Driving Scenarioscited by 63
  24. 2024A Survey for Foundation Models in Autonomous Drivingcited by 51
  25. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertisecited by 49
  26. 2019A Comprehensive Benchmark for Single Image Compression Artifacts Reductioncited by 31
  27. 2024ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Modelscited by 26
  28. 2023Open-sourced Data Ecosystem in Autonomous Driving: the Present and Futurecited by 22
  29. 2023HQ-50K: A Large-scale, High-quality Dataset for Image Restorationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 125; 2026: 145.

284 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2025Qwen2.5-Omni Technical Reportcited by 704
  5. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  6. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  7. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  8. 2025Thyme: Think Beyond Imagescited by 160
  9. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  10. 2026Qwen3.5-Omni Technical Reportcited by 146
  11. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  12. 2025Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Searchcited by 116
38 more
  1. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 111
  2. 2025MiMo-VL Technical Reportcited by 106
  3. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 99
  4. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  5. 2025Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMscited by 94
  6. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  7. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  8. 2025Monet: Reasoning in Latent Visual Space Beyond Images and Languagecited by 75
  9. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  10. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 71
  11. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 67
  12. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  13. 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokenscited by 65
  14. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
  15. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
  16. 2024GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Taskscited by 50
  17. 2025XLRS-Bench: Could Your Multimodal LLMs Understand Extremely Large Ultra-High-Resolution Remote Sensing Imagery?cited by 50
  18. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 47
  19. 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learningcited by 46
  20. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMscited by 41
  21. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  22. 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37
  23. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  24. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  25. 2025SimScale: Learning to Drive via Real-World Simulation at Scalecited by 34
  26. 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 31
  27. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 26
  28. 2025SITE: towards Spatial Intelligence Thorough Evaluationcited by 26
  29. 2025Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearchcited by 25
  30. 2025MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Modelscited by 24
  31. 2025Sketch-in-Latents: Eliciting Unified Reasoning in MLLMscited by 24
  32. 2025Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Toolscited by 23
  33. 2025Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAGcited by 22
  34. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipescited by 22
  35. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decodingcited by 21
  36. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  37. 2025FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registerscited by 20
  38. 2025FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answeringcited by 20

50 most cited of 284 All 284 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models
  2. 2024MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks
  3. 2024MileBench: Benchmarking MLLMs in Long Context
  4. 2026Multimodal Large Language Models as Image Classifiers
  5. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  6. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  7. 2026What We are Missing in Multimodal LLM Evaluation?
  8. 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark