mlsanityData to 14 Oct 2026

MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding

arXiv 2508.15802 ↗First posted 14 Aug 2025cs.CL cs.AIcited by 12

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

As multimodal large language models (MLLMs) grow increasingly capable, fixed benchmarks are gradually losing their effectiveness in evaluating high-level scientific understanding. In this paper, we introduce the Multimodal Academic Cover benchmark (MAC), a live benchmark that could continuously evolve with scientific advancement and model progress. MAC leverages over 25,000 image-text pairs sourced from issues of top-tier scientific journals such as Nature, Science, and Cell, challenging MLLMs to reason across abstract visual and textual scientific content. Experiments on our most recent yearly snapshot, MAC-2025, reveal that while MLLMs demonstrate strong perceptual abilities, their cross-modal scientific reasoning remains limited. To bridge this gap, we propose DAD, a lightweight inference-time approach that enhances MLLMs by extending MLLM visual features with language space reasoning, achieving performance improvements of up to 11%. Finally, we highlight the live nature of MAC through experiments on updating journal covers and models for curation, illustrating its potential to remain aligned with the frontier of human knowledge. We release our benchmark at https://github.com/mhjiang0408/MAC_Bench.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 59 references that are in the map. First posted on arXiv 14 Aug 2025.

Builds on

42 of 59 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models17 Feb
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  3. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models5 Aug
  4. 2024MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model17 Jun
  5. 2024What is the Visual Cognition Gap between Humans and Multimodal LLMs?14 Jun
  6. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding13 Jun
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2017On Calibration of Modern Neural Networkscited by 4,960
  9. 2024GPT-4o System Cardcited by 4,588
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
23 more
  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  5. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  6. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  8. 2018Deep Anomaly Detection with Outlier Exposurecited by 942
  9. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  10. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  11. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  12. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  13. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  14. 2019PlotQA: Reasoning over Scientific Plotscited by 321
  15. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
  16. 2023Data Filtering Networkscited by 250
  17. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 234
  18. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 139
  19. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 110
  20. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
  21. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 68
  22. 2023mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Modelcited by 32
  23. 2025Science-T2I: Addressing Scientific Illusions in Image Synthesiscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 9.

12 papers.

  1. 2025DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesiscited by 34
  2. 2025InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Researchcited by 19
  3. 2026OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Modelscited by 4
  4. 2026The Side Effects of Being Smart: Safety Risks in MLLMs' Multi-Image Reasoningcited by 3
  5. 2026Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theorycited by 3
  6. 2025ImageNet-Think-250K: A Large-Scale Synthetic Dataset for Multimodal Reasoning for Vision Language Modelscited by 1
  7. 2026BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QAcited by 1
  8. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  9. 2026MMBench-Live: A Continuously Evolving Benchmark for Multimodal Modelscited by 0
  10. 2026MatPhaseBench: A Semantics-Guided Benchmark for Materials Phase Diagrams Understandingcited by 0
  11. 2026An Exam for Active Observerscited by 0
  12. 2026A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Imagescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems
  2. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension
  3. 2026WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark
  4. 2025MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models
  5. 2025SO-Bench: A Structural Output Evaluation of Multimodal LLMs
  6. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  7. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
  8. 2023SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructions