mlsanityData to 7 Oct 2026

Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning

arXiv 2506.10521 ↗First posted 12 Jun 2025cs.AI cs.CLcited by 33

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Scientific discoveries increasingly rely on complex multimodal reasoning based on information-intensive scientific data and domain-specific expertise. Empowered by expert-level scientific benchmarks, scientific Multimodal Large Language Models (MLLMs) hold the potential to significantly enhance this discovery process in realistic workflows. However, current scientific benchmarks mostly focus on evaluating the knowledge understanding capabilities of MLLMs, leading to an inadequate assessment of their perception and reasoning abilities. To address this gap, we present the Scientists' First Exam (SFE) benchmark, designed to evaluate the scientific cognitive capacities of MLLMs through three interconnected levels: scientific signal perception, scientific attribute understanding, scientific comparative reasoning. Specifically, SFE comprises 830 expert-verified VQA pairs across three question types, spanning 66 multimodal tasks across five high-value disciplines. Extensive experiments reveal that current state-of-the-art GPT-o3 and InternVL-3 achieve only 34.08% and 26.52% on SFE, highlighting significant room for MLLMs to improve in scientific realms. We hope the insights obtained in SFE will facilitate further developments in AI-enhanced scientific discoveries.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 87 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

37 of 87 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec

Other references · 36

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2025Qwen2.5-VL Technical Reportcited by 5,418
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  10. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  11. 2024A Survey on LLM-as-a-Judgecited by 1,384
  12. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
24 more
  1. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  2. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  3. 2025Humanity's Last Examcited by 467
  4. 2025Seed1.5-VL Technical Reportcited by 367
  5. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  6. 2020ProGen: Language Modeling for Protein Generationcited by 127
  7. 2024ChemLLM: A Chemical Large Language Modelcited by 100
  8. 2023BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicinecited by 99
  9. 2021ClimateBert: A Pretrained Language Model for Climate-Related Textcited by 89
  10. 2023InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discoverycited by 86
  11. 2023MolXPT: Wrapping Molecules with Text for Generative Pre-trainingcited by 81
  12. 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplinescited by 67
  13. 2024CasCast: Skillful High-resolution Precipitation Nowcasting via Cascaded Modellingcited by 52
  14. 2023OceanGPT: A Large Language Model for Ocean Science Taskscited by 42
  15. 2023DrugChat: Towards Enabling ChatGPT-Like Capabilities on Drug Molecule Graphscited by 40
  16. 2023GeoGalactica: A Scientific Large Language Model in Geosciencecited by 32
  17. 2024MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Predictioncited by 24
  18. 2024ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Trainingcited by 21
  19. 2024WeatherQA: Can Multimodal Language Models Reason about Severe Weather?cited by 21
  20. 2024CLLMate: A Multimodal Benchmark for Weather and Climate Events Forecastingcited by 19
  21. 2024DrugLLM: Open Large Language Model for Few-shot Molecule Generationcited by 15
  22. 2021ELECTRAMed: a new pre-trained language representation model for biomedical NLPcited by 11
  23. 2025EarthGPT-X: A Spatial MLLM for Multi-level Multi-Source Remote Sensing Imagery Understanding with Visual Promptingcited by 11
  24. 2024PharmaGPT: Domain-Specific Large Language Models for Bio-Pharmaceutical and Chemistrycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 17.

33 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  2. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  3. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 49
  4. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
  5. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  6. 2026InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editingcited by 23
  7. 2025PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Levelcited by 14
  8. 2025AutoLabs: Cognitive Multi-Agent Systems with Self-Correction for Autonomous Chemical Experimentationcited by 13
  9. 2025SciTS: Scientific Time Series Understanding and Generation with LLMscited by 13
  10. 2026SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agentscited by 12
  11. 2026GeoReason: Aligning Thinking And Answering In Remote Sensing Vision-Language Models Via Logical Consistency Reinforcement Learningcited by 10
  12. 2025SciReasoner: Laying the Scientific Reasoning Ground Across Disciplinescited by 9
21 more
  1. 2025GEMeX-RMCoT: An Enhanced Med-VQA Dataset for Region-Aware Multimodal Chain-of-Thought Reasoningcited by 8
  2. 2025RadarQA: Multi-modal Quality Analysis of Weather Radar Forecastscited by 8
  3. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  4. 2026Innovator-VL: A Multimodal Large Language Model for Scientific Discoverycited by 7
  5. 2025ExpVid: A Benchmark for Experiment Video Understanding & Reasoningcited by 4
  6. 2026When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoningcited by 4
  7. 2025SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligencecited by 3
  8. 2025HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discoverycited by 3
  9. 2026MolEmb: Multimodal Large Language Models Can Be Strong Molecular Embedding Modelscited by 3
  10. 2025ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflowscited by 2
  11. 2026SciDataCopilot: An Agentic Data Preparation Framework for AGI-driven Scientific Discoverycited by 2
  12. 2026S1-Omni-Image: A Unified Model for Scientific Image Understanding, Generation, and Editingcited by 2
  13. 2026Scalable Visual Pretraining for Language Intelligencecited by 2
  14. 2026TopoAgent: A Self-Evolving Topological Agent for Multimodal Scientific Reasoningcited by 2
  15. 2026SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integrationcited by 2
  16. 2025Bidirectional Representations Augmented Autoregressive Biological Sequence Generationcited by 1
  17. 2026PolyReal: A Benchmark for Real-World Polymer Science Workflowscited by 1
  18. 2025SynWeather: Weather Observation Data Synthesis across Multiple Regions and Variables via a General Diffusion Transformercited by 0
  19. 2026AblateCell: A Reproduce-then-Ablate Agent for Virtual Cell Repositoriescited by 0
  20. 2026SciOrch: Learning to Orchestrate Expert LLMs for Solving Frontier Multimodal Scientific Reasoning Taskscited by 0
  21. 2026MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning
  2. 2026SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation
  3. 2024VisScience: An Extensive Benchmark for Evaluating K12 Educational Multi-modal Scientific Reasoning
  4. 2025CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMs
  5. 2025MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams
  6. 2025SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problems
  7. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
  8. 2025MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models