mlsanityData to 14 Oct 2026

Large Language Models Achieve Gold Medal Performance at the International Olympiad on Astronomy & Astrophysics (IOAA)

arXiv 2510.05016 ↗First posted 6 Oct 2025astro-ph.IM cs.AI cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

While task-specific demonstrations show early success in applying large language models (LLMs) to automate some astronomical research tasks, they only provide incomplete views of all necessary capabilities in solving astronomy problems, calling for more thorough understanding of LLMs' strengths and limitations. So far, existing benchmarks and evaluations focus on simple question-answering that primarily tests astronomical knowledge and fails to evaluate the complex reasoning required for real-world research in the discipline. Here, we address this gap by systematically benchmarking five state-of-the-art LLMs on the International Olympiad on Astronomy and Astrophysics (IOAA) exams, which are designed to examine deep conceptual understanding, multi-step derivations, and multimodal analysis. With average scores of 85.6% and 84.2%, Gemini 2.5 Pro and GPT-5 (the two top-performing models) not only achieve gold medal level performance but also rank in the top two among ~200-300 participants in all four IOAA theory exams evaluated (2022-2025). In comparison, results on the data analysis exams show more divergence. GPT-5 still excels in the exams with an 88.5% average score, ranking top 10 among the participants in the four most recent IOAAs, while other models' performances drop to 48-76%. Furthermore, our in-depth error analysis underscores conceptual reasoning, geometric reasoning, and spatial visualization (52-79% accuracy) as consistent weaknesses among all LLMs. Hence, although LLMs approach peak human performance in theory exams, critical gaps must be addressed before they can serve as autonomous research agents in astronomy.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 12 of its 37 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

12 of 37 references.

References · 12

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2025Accelerating scientific discovery with Co-Scientistcited by 401
  3. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  4. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 234
  5. 2025Goedel-Prover: A Frontier Model for Open-Source Automated Theorem Provingcited by 133
  6. 2025Competitive Programming with Large Reasoning Modelscited by 119
  7. 2025Effective Training Data Synthesis for Improving MLLM Chart Understandingcited by 20
  8. 2025Physics Supernova: AI Agent Matches Elite Gold Medalists at IPhO 2025cited by 14
  9. 2021A Deep Learning Approach for Active Anomaly Detection of Extragalactic Transientscited by 11
  10. 2025Automated Algorithmic Discovery for Scientific Computing through LLM-Guided Evolutionary Search: A Case Study in Gravitational-Wave Detectioncited by 9
  11. 2024AstroMLab 1: Who Wins Astronomy Jeopardy!?cited by 7
  12. 2025AstroMLab 4: Benchmark-Topping Performance in Astronomy Q&A with a 70B-Parameter Domain-Specialized Reasoning Modelcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2024Designing an Evaluation Framework for Large Language Models in Astronomy Research
  2. 2025AstroMMBench: A Benchmark for Evaluating Multimodal Large Language Models Capabilities in Astronomy
  3. 2025From Queries to Criteria: Understanding How Astronomers Evaluate LLMs
  4. 2023Harnessing the Power of Adversarial Prompting and Large Language Models for Robust Hypothesis Generation in Astronomy
  5. 2024AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets
  6. 2026AstroAlertBench: Evaluating the Accuracy, Reasoning, and Honesty of Multimodal LLMs in Astronomical Classification
  7. 2026Taramandal-GPT: Enhancing Astrodynamics Problem-Solving with Knowledge Retrieval and Structured Thinking
  8. 2026Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy