mlsanityData to 7 Oct 2026

MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks

arXiv 2410.10563 ↗First posted 14 Oct 2024cs.CVcited by 41

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present MEGA-Bench, an evaluation suite that scales multimodal evaluation to over 500 real-world tasks, to address the highly heterogeneous daily use cases of end users. Our objective is to optimize for a set of high-quality data samples that cover a highly diverse and rich set of multimodal tasks, while enabling cost-effective and accurate model evaluation. In particular, we collected 505 realistic tasks encompassing over 8,000 samples from 16 expert annotators to extensively cover the multimodal task space. Instead of unifying these problems into standard multi-choice questions (like MMMU, MMBench, and MMT-Bench), we embrace a wide range of output formats like numbers, phrases, code, LaTeX, coordinates, JSON, free-form, etc. To accommodate these formats, we developed over 40 metrics to evaluate these tasks. Unlike existing benchmarks, MEGA-Bench offers a fine-grained capability report across multiple dimensions (e.g., application, input type, output format, skill), allowing users to interact with and visualize model capabilities in depth. We evaluate a wide variety of frontier vision-language models on MEGA-Bench to understand their capabilities across these dimensions.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Oct 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 26; 2026: 12.

41 papers.

Sort:

By area:

  1. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  2. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  3. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  4. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  5. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  6. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Searchcited by 34
  7. 2025MM-CoT:A Benchmark for Probing Visual Chain-of-Thought Reasoning in Multimodal Modelscited by 33
  8. 2025One RL to See Them All: Visual Triple Unified Reinforcement Learningcited by 29
  9. 2024SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Modelscited by 28
  10. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 28
  11. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  12. 2025TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibilitycited by 16
29 more
  1. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  2. 2024POINTS1.5: Building a Vision-Language Model towards Real World Applicationscited by 12
  3. 2025VFaith: Do Large Multimodal Models Really Reason on Seen Images Rather than Previous Memories?cited by 7
  4. 2026Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoningcited by 6
  5. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  6. 2025Limits and Gains of Test-Time Scaling in Vision-Language Reasoningcited by 5
  7. 2025ADIEE: Automatic Dataset Creation and Scorer for Instruction-Guided Image Editing Evaluationcited by 4
  8. 2025Draw-In-Mind: Rebalancing Designer-Painter Roles in Unified Multimodal Models Benefits Image Editingcited by 4
  9. 2025Metis-SPECS: Decoupling Multimodal Learning via Self-distilled Preference-based Cold Startcited by 4
  10. 2025RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answeringcited by 4
  11. 2026RegionReasoner: Region-Grounded Multi-Round Visual Reasoningcited by 4
  12. 2025PixelWorld: How Far Are We from Perceiving Everything as Pixels?cited by 3
  13. 2025MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planningcited by 3
  14. 2025MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessmentcited by 3
  15. 2026WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmarkcited by 3
  16. 2025MOAT: Evaluating LMMs for Capability Integration and Instruction Groundingcited by 2
  17. 2025VoQA: Visual-only Question Answeringcited by 2
  18. 2026GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documentscited by 2
  19. 2025OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Taskscited by 1
  20. 2025Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scalecited by 1
  21. 2025Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Modelscited by 1
  22. 2026Automatic Cognitive Task Generation for In-Situ Evaluation of Embodied Agentscited by 1
  23. 2025Prompt-to-Parts: Generative AI for Physical Assembly and Scalable Instructionscited by 0
  24. 2026Perception-Aware Multimodal Spatial Reasoning from Monocular Imagescited by 0
  25. 2026KVCapsule: Efficient Sequential KV Cache Compression for Vision-Language Models with Asymmetric Redundancycited by 0
  26. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  27. 2026VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrievalcited by 0
  28. 2026Taming CoT Obfuscation in VLMs: From Mechanistic Evidence to Activation Enforcementcited by 0
  29. 2026Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  2. 2024Automatic benchmarking of large multimodal models via iterative experiment programming
  3. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?
  4. 2024Task Me Anything
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
  6. 2025SO-Bench: A Structural Output Evaluation of Multimodal LLMs
  7. 2024MileBench: Benchmarking MLLMs in Long Context
  8. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks