mlsanityData to 7 Oct 2026

AstroMMBench: A Benchmark for Evaluating Multimodal Large Language Models Capabilities in Astronomy

arXiv 2510.00063 ↗First posted 29 Sep 2025astro-ph.IM cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Astronomical image interpretation presents a significant challenge for applying multimodal large language models (MLLMs) to specialized scientific tasks. Existing benchmarks focus on general multimodal capabilities but fail to capture the complexity of astronomical data. To bridge this gap, we introduce AstroMMBench, the first comprehensive benchmark designed to evaluate MLLMs in astronomical image understanding. AstroMMBench comprises 621 multiple-choice questions across six astrophysical subfields, curated and reviewed by 15 domain experts for quality and relevance. We conducted an extensive evaluation of 25 diverse MLLMs, including 22 open-source and 3 closed-source models, using AstroMMBench. The results show that Ovis2-34B achieved the highest overall accuracy (70.5%), demonstrating leading capabilities even compared to strong closed-source models. Performance showed variations across the six astrophysical subfields, proving particularly challenging in domains like cosmology and high-energy astrophysics, while models performed relatively better in others, such as instrumentation and solar astrophysics. These findings underscore the vital role of domain-specific benchmarks like AstroMMBench in critically evaluating MLLM performance and guiding their targeted development for scientific applications. AstroMMBench provides a foundational resource and a dynamic tool to catalyze advancements at the intersection of AI and astronomy.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 55 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

34 of 55 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
22 more
  1. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  5. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  6. 2024DeepSeek-V3 Technical Reportcited by 724
  7. 2024Yi: Open Foundation Models by 01.AIcited by 706
  8. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  9. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  10. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  11. 2024InternLM2 Technical Reportcited by 391
  12. 2025Kimi-VL Technical Reportcited by 381
  13. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  14. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  15. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  16. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  17. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 127
  18. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  19. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  20. 2024AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perceptioncited by 63
  21. 2025Scalable Vision Language Model Training via High Quality Data Curationcited by 54
  22. 2024MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation modelscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraintscited by 2
  2. 2026SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopycited by 0
  3. 2026Generalist Vision-Language Models for Fast Radio Burst detection: a zero-shot benchmark against a specialized detectorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AstroAlertBench: Evaluating the Accuracy, Reasoning, and Honesty of Multimodal LLMs in Astronomical Classification
  2. 2026A systematic evaluation of vision-language models for observational astronomical reasoning tasks
  3. 2025Large Language Models Achieve Gold Medal Performance at the International Olympiad on Astronomy & Astrophysics (IOAA)
  4. 2025AstroMLab 4: Benchmark-Topping Performance in Astronomy Q&A with a 70B-Parameter Domain-Specialized Reasoning Model
  5. 2025AstroLLaVA: towards the unification of astronomical data and natural language
  6. 2025Radio Astronomy in the Era of Vision-Language Models: Prompt Sensitivity and Adaptation
  7. 2024At First Sight: Zero-Shot Classification of Astronomical Images with Large Multimodal Models
  8. 2024Designing an Evaluation Framework for Large Language Models in Astronomy Research