# MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks

arXiv: https://arxiv.org/abs/2410.10563 · HTML: https://mlsanity.com/paper/2410.10563/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2024.

## Built on by (most cited first, 30 of 41)

- [VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning](https://mlsanity.com/paper/2504.08837/) (2025)
- [Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning](https://mlsanity.com/paper/2505.15966/) (2025)
- [Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers](https://mlsanity.com/paper/2506.23918/) (2025)
- [MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale](https://mlsanity.com/paper/2412.05237/) (2024)
- [NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation](https://mlsanity.com/paper/2504.13055/) (2025)
- [VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search](https://mlsanity.com/paper/2503.10582/) (2025)
- [MM-CoT:A Benchmark for Probing Visual Chain-of-Thought Reasoning in Multimodal Models](https://mlsanity.com/paper/2512.08228/) (2025)
- [One RL to See Them All: Visual Triple Unified Reinforcement Learning](https://mlsanity.com/paper/2505.18129/) (2025)
- [SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models](https://mlsanity.com/paper/2411.13112/) (2024)
- [Holistic Evaluation of Multimodal LLMs on Spatial Intelligence](https://mlsanity.com/paper/2508.13142/) (2025)
- [Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks](https://mlsanity.com/paper/2504.18838/) (2025)
- [TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibility](https://mlsanity.com/paper/2510.07550/) (2025)
- [MMPB: It's Time for Multi-Modal Personalization](https://mlsanity.com/paper/2509.22820/) (2025)
- [POINTS1.5: Building a Vision-Language Model towards Real World Applications](https://mlsanity.com/paper/2412.08443/) (2024)
- [VFaith: Do Large Multimodal Models Really Reason on Seen Images Rather than Previous Memories?](https://mlsanity.com/paper/2506.11571/) (2025)
- [Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding](https://mlsanity.com/paper/2603.18472/) (2026)
- [Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoning](https://mlsanity.com/paper/2602.11455/) (2026)
- [Metis-SPECS: Decoupling Multimodal Learning via Self-distilled Preference-based Cold Start](https://mlsanity.com/paper/2510.25801/) (2025)
- [Limits and Gains of Test-Time Scaling in Vision-Language Reasoning](https://mlsanity.com/paper/2512.11109/) (2025)
- [PixelWorld: How Far Are We from Perceiving Everything as Pixels?](https://mlsanity.com/paper/2501.19339/) (2025)
- [ADIEE: Automatic Dataset Creation and Scorer for Instruction-Guided Image Editing Evaluation](https://mlsanity.com/paper/2507.07317/) (2025)
- [Draw-In-Mind: Rebalancing Designer-Painter Roles in Unified Multimodal Models Benefits Image Editing](https://mlsanity.com/paper/2509.01986/) (2025)
- [RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering](https://mlsanity.com/paper/2512.05119/) (2025)
- [RegionReasoner: Region-Grounded Multi-Round Visual Reasoning](https://mlsanity.com/paper/2602.03733/) (2026)
- [WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark](https://mlsanity.com/paper/2606.06538/) (2026)
- [MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning](https://mlsanity.com/paper/2506.22992/) (2025)
- [MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessment](https://mlsanity.com/paper/2508.17290/) (2025)
- [MOAT: Evaluating LMMs for Capability Integration and Instruction Grounding](https://mlsanity.com/paper/2503.09348/) (2025)
- [VoQA: Visual-only Question Answering](https://mlsanity.com/paper/2505.14227/) (2025)
- [OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Tasks](https://mlsanity.com/paper/2505.18775/) (2025)

## Abstract (arXiv)

We present MEGA-Bench, an evaluation suite that scales multimodal evaluation to over 500 real-world tasks, to address the highly heterogeneous daily use cases of end users. Our objective is to optimize for a set of high-quality data samples that cover a highly diverse and rich set of multimodal tasks, while enabling cost-effective and accurate model evaluation. In particular, we collected 505 realistic tasks encompassing over 8,000 samples from 16 expert annotators to extensively cover the multimodal task space. Instead of unifying these problems into standard multi-choice questions (like MMMU, MMBench, and MMT-Bench), we embrace a wide range of output formats like numbers, phrases, code, LaTeX, coordinates, JSON, free-form, etc. To accommodate these formats, we developed over 40 metrics to evaluate these tasks. Unlike existing benchmarks, MEGA-Bench offers a fine-grained capability report across multiple dimensions (e.g., application, input type, output format, skill), allowing users to interact with and visualize model capabilities in depth. We evaluate a wide variety of frontier vision-language models on MEGA-Bench to understand their capabilities across these dimensions.