mlsanityData to 14 Oct 2026

JMMMU-Pro: Image-based Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Construction

arXiv 2512.14620 ↗First posted 16 Dec 2025cs.CL cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper introduces JMMMU-Pro, an image-based Japanese Multi-discipline Multimodal Understanding Benchmark, and Vibe Benchmark Construction, a scalable construction method. Following the evolution from MMMU to MMMU-Pro, JMMMU-Pro extends JMMMU by composing the question image and question text into a single image, thereby creating a benchmark that requires integrated visual-textual understanding through visual perception. To build JMMMU-Pro, we propose Vibe Benchmark Construction, a methodology in which an image generative model (e.g., Nano Banana Pro) produces candidate visual questions, and humans verify the outputs and, when necessary, regenerate with adjusted prompts to ensure quality. By leveraging Nano Banana Pro's highly realistic image generation capabilities and its ability to embed clean Japanese text, we construct a high-quality benchmark at low cost, covering a wide range of background and layout designs. Experimental results show that all open-source LMMs struggle substantially with JMMMU-Pro, underscoring JMMMU-Pro as an important benchmark for guiding future efforts in the open-source community. We believe that JMMMU-Pro provides a more rigorous evaluation tool for assessing the Japanese capabilities of LMMs and that our Vibe Benchmark Construction also offers an efficient guideline for future development of image-based VQA benchmarks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 44 of its 53 references that are in the map. First posted on arXiv 16 Dec 2025.

Builds on

44 of 53 references.

References · 44

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  6. 2025Qwen3-VL Technical Reportcited by 2,793
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  10. 2024Qwen2 Technical Reportcited by 2,120
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  12. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
32 more
  1. 2019Towards VQA Models That Can Readcited by 1,718
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  8. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  9. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  10. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  11. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  12. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  13. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  14. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 244
  15. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  16. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  17. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 144
  18. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 64
  19. 2025Aya Vision: Advancing the Frontier of Multilingual Multimodalitycited by 33
  20. 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probingcited by 31
  21. 2024VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Modelcited by 26
  22. 2024JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluationcited by 19
  23. 2025Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absentcited by 15
  24. 2024Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Modelscited by 13
  25. 2023MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Modelscited by 12
  26. 2024From Prompt Engineering to Prompt Science With Human in the Loopcited by 12
  27. 2024Heron-Bench: A Benchmark for Evaluating Vision Language Models in Japanesecited by 12
  28. 2024JDocQA: Japanese Document Question Answering Dataset for Generative Language Modelscited by 9
  29. 2025MangaVQA and MangaLMM: A Benchmark and Specialized Model for Multimodal Manga Understandingcited by 7
  30. 2024Constructing Multimodal Datasets from Scratch for Rapid Development of a Japanese Visual Language Modelcited by 6
  31. 2025LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?cited by 4
  32. 2025Harnessing PDF Data for Improving Japanese Large Multimodal Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation
  2. 2026Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models
  3. 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark
  4. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  5. 2026JaWildText: A Benchmark for Vision-Language Models on Japanese Scene Text Understanding
  6. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Models
  7. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension
  8. 2024MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspective