mlsanityData to 14 Oct 2026

JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation

arXiv 2410.17250 ↗First posted 22 Oct 2024cs.CL cs.AI cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Accelerating research on Large Multimodal Models (LMMs) in non-English languages is crucial for enhancing user experiences across broader populations. In this paper, we introduce JMMMU (Japanese MMMU), the first large-scale Japanese benchmark designed to evaluate LMMs on expert-level tasks based on the Japanese cultural context. To facilitate comprehensive culture-aware evaluation, JMMMU features two complementary subsets: (i) culture-agnostic (CA) subset, where the culture-independent subjects (e.g., Math) are selected and translated into Japanese, enabling one-to-one comparison with its English counterpart MMMU; and (ii) culture-specific (CS) subset, comprising newly crafted subjects that reflect Japanese cultural context. Using the CA subset, we observe performance drop in many LMMs when evaluated in Japanese, which is purely attributable to language variation. Using the CS subset, we reveal their inadequate Japanese cultural understanding. Further, by combining both subsets, we identify that some LMMs perform well on the CA subset but not on the CS subset, exposing a shallow understanding of the Japanese language that lacks depth in cultural understanding. We hope this work will not only help advance LMM performance in Japanese but also serve as a guideline to create high-standard, culturally diverse benchmarks for multilingual LMM development. The project page is https://mmmu-japanese-benchmark.github.io/JMMMU/.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 46 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

28 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024CogVLM2: Visual Language Models for Image and Video Understanding29 Aug
  2. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark10 Jun
  3. 2024Heron-Bench: A Benchmark for Evaluating Vision Language Models in Japanese11 Apr
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 24

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  5. 2015VQA: Visual Question Answeringcited by 2,506
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  11. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  12. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
12 more
  1. 2023Larger language models do in-context learning differentlycited by 394
  2. 2024Evolutionary Optimization of Model Merging Recipescited by 208
  3. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  4. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  5. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  6. 2015Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answeringcited by 80
  7. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  8. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 64
  9. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  10. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  11. 2023DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documentscited by 7
  12. 2024FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe Generationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 8.

19 papers.

Sort:

By area:

  1. 2025CARE: Multilingual Human Preference Learning for Cultural Awarenesscited by 15
  2. 2025MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMscited by 13
  3. 2025DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culturecited by 13
  4. 2025CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMscited by 12
  5. 2025Pearl: A Multimodal Culturally-Aware Arabic Instruction Datasetcited by 6
  6. 2026Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective Responsescited by 6
  7. 2025VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwancited by 3
  8. 2025MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Languagecited by 3
  9. 2025MyCulture: Exploring Malaysia's Diverse Culture under Low-Resource Language Constraintscited by 3
  10. 2024Development of a Large-scale Dataset of Chest Computed Tomography Reports in Japanese and a High-performance Finding Classification Modelcited by 1
  11. 2025Harnessing PDF Data for Improving Japanese Large Multimodal Modelscited by 1
  12. 2026Defining Cultural Capabilities for AI Evaluation: A Taxonomy Grounded in Intercultural Communication Theorycited by 1
7 more
  1. 2025JMMMU-Pro: Image-based Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Constructioncited by 0
  2. 2026JaWildText: A Benchmark for Vision-Language Models on Japanese Scene Text Understandingcited by 0
  3. 2026Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Modelscited by 0
  4. 2026KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Contextcited by 0
  5. 2026Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameterscited by 0
  6. 2026MET: Theory-Grounded and Culture-Aware Multilingual Moral Reasoningcited by 0
  7. 2026CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation
  2. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
  3. 2025Grounding Multilingual Multimodal LLMs With Cultural Knowledge
  4. 2026CrossCult-KIBench: A Benchmark for Cross-Cultural Knowledge Insertion in MLLMs
  5. 2024M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models
  6. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages
  7. 2026Human-Grounded Multimodal Benchmark with 900K-Scale Aggregated Student Response Distributions from Japan's National Assessment of Academic Ability
  8. 2025MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models