AstroMLab 4: Benchmark-Topping Performance in Astronomy Q&A with a 70B-Parameter Domain-Specialized Reasoning Model
On the map: LLM reasoning → Knowledge- and tool-augmented reasoning
Abstract (arXiv)
General-purpose large language models (LLMs), despite their broad capabilities, often struggle with specialized domain knowledge. This gap hinders their deployment as reliable research agents in demanding fields such as astronomy. Building on our prior work with AstroSage-Llama-3.1-8B, this study introduces AstroSage-Llama-3.1-70B, a 70-billion parameter domain-specialized natural-language AI assistant. It is designed for research and education across astronomy, astrophysics, space science, astroparticle physics, cosmology, and astronomical instrumentation. Developed from the Meta-Llama-3.1-70B foundation, AstroSage-Llama-3.1-70B underwent extensive continued pre-training (CPT) on a vast corpus of astronomical literature, followed by supervised fine-tuning (SFT) and model merging. We integrated reasoning chains into the SFT dataset, enabling AstroSage-Llama-3.1-70B to either answer the user query immediately, or first emit a human-readable thought process. Evaluated on a validated subset of 3,846 questions from the AstroMLab-1 benchmark (Ting et al., 2024) -- derived from literature withheld during training -- AstroSage-Llama-3.1-70B achieves top-tier performance (89.0%), matching GPT-5.2, Claude-4.5-Opus, and Gemini-3-Pro while being more cost-efficient. This work demonstrates that domain specialization, when applied to large-scale models, can enable them to outperform generalist counterparts in specialized knowledge areas like astronomy, thereby advancing the frontier of AI capabilities in the field.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 6 of its 10 references that are in the map. First posted on arXiv 23 May 2025.
Builds on
6 of 10 references.
References · 6
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
- 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
- 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
- 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
- 2023Optimizing Distributed Training on Frontier for Large Language Modelscited by 12
Built on by
Citing papers per year: 2025: 4; 2026: 3.
7 papers.
- 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
- 2025Large Language Models Achieve Gold Medal Performance at the International Olympiad on Astronomy & Astrophysics (IOAA)cited by 3
- 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
- 2025AstroReview: An LLM-driven Multi-Agent Framework for Telescope Proposal Peer Review and Refinementcited by 0
- 2026Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branchescited by 0
- 2026Rethinking Domain Specialization for Open-Ended Scientific Reasoning in Astronomy Language Modelscited by 0
- 2026Taramandal-GPT: Enhancing Astrodynamics Problem-Solving with Knowledge Retrieval and Structured Thinkingcited by 0
- 2026Taramandal-GPT: Enhancing Astrodynamics Problem-Solving with Knowledge Retrieval and Structured Thinking21 Sep 2026
- 2026Rethinking Domain Specialization for Open-Ended Scientific Reasoning in Astronomy Language Models15 Sep 2026
- 2026Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branches31 May 2026
- 2025AstroReview: An LLM-driven Multi-Agent Framework for Telescope Proposal Peer Review and Refinement31 Dec 2025
- 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents20 Oct 2025
- 2025Large Language Models Achieve Gold Medal Performance at the International Olympiad on Astronomy & Astrophysics (IOAA)6 Oct 2025
- 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers28 Aug 2025
Similar, not cited
Close in wording, no citation either way
- 2025AstroMMBench: A Benchmark for Evaluating Multimodal Large Language Models Capabilities in Astronomy
- 2024AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets
- 2026AstroAlertBench: Evaluating the Accuracy, Reasoning, and Honesty of Multimodal LLMs in Astronomical Classification
- 2024Designing an Evaluation Framework for Large Language Models in Astronomy Research
- 2025AstroLLaVA: towards the unification of astronomical data and natural language
- 2026A systematic evaluation of vision-language models for observational astronomical reasoning tasks
- 2026Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy
- 2023Harnessing the Power of Adversarial Prompting and Large Language Models for Robust Hypothesis Generation in Astronomy