mlsanityData to 14 Oct 2026

ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks

arXiv 2503.06885 ↗First posted 10 Mar 2025cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Solving expert-level multimodal tasks is a key milestone towards general intelligence. As the capabilities of multimodal large language models (MLLMs) continue to improve, evaluation of such advanced multimodal intelligence becomes necessary yet challenging. In this work, we introduce ProBench, a benchmark of open-ended user queries that require professional expertise and advanced reasoning. ProBench consists of 4,000 high-quality samples independently submitted by professionals based on their daily productivity demands. It spans across 10 fields and 56 sub-fields, including science, arts, humanities, coding, mathematics, and creative writing. Experimentally, we evaluate and compare 24 latest models using MLLM-as-a-Judge. Our results reveal that although the best open-source models rival the proprietary ones, ProBench presents significant challenges in visual perception, textual understanding, domain knowledge and advanced reasoning, thus providing valuable directions for future multimodal AI research efforts.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

23 of 36 references.

References · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2019Towards VQA Models That Can Readcited by 1,718
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
11 more
  1. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  2. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  3. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  4. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  5. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  6. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  7. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 138
  8. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  9. 2025CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingscited by 115
  10. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  11. 2024VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulationcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  2. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  2. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
  3. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning
  4. 2025MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Models
  5. 2025MANBench: Is Your Multimodal Model Smarter than Human?
  6. 2025MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams
  7. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  8. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria