Rethinking Domain Specialization for Open-Ended Scientific Reasoning in Astronomy Language Models
On the map: LLM reasoning → Knowledge- and tool-augmented reasoning
Abstract (arXiv)
Domain-specialized language models are widely used for scientific question answering, but stronger general-purpose systems raise a sharper question: when does domain-specific fine-tuning remain valuable for open-ended scientific reasoning? We study this in astronomy with a curated QA benchmark from publicly available 2017--2026 Olympiad-style materials. The free-response subset contains 300 questions, including 204 text-only and 96 image-linked examples. We compare open-weight and API-served general-purpose, multimodal, and astronomy-specialized models using judge-based correctness and complementary reference metrics. Strong general-purpose models establish the highest correctness baseline in this testbed, while analyses of metric agreement, judge sensitivity, benchmark composition, and modality reveal variation not captured by a single leaderboard. These results motivate treating domain specialization as a task- and deployment-dependent property and highlight the role of domain-specific evaluation in determining which models, capabilities, and evaluation criteria are appropriate for scientific workflows.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 21 references that are in the map. First posted on arXiv 15 Sep 2026.
Builds on
17 of 21 references.
Closest ancestors · 1
Same direction, newest first
Other references · 16
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
- 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
- 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
- 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
- 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136
4 more
- 2023AstroLLaMA: Towards Specialized Foundation Models in Astronomycited by 41
- 2024AstroMLab 1: Who Wins Astronomy Jeopardy!?cited by 7
- 2024AstroMLab 2: AstroLLaMA-2-70B Model and Benchmarking Specialised LLMs for Astronomycited by 6
- 2025EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistantscited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Taramandal-GPT: Enhancing Astrodynamics Problem-Solving with Knowledge Retrieval and Structured Thinking
- 2025DomainCQA: Crafting Knowledge-Intensive QA from Domain-Specific Charts
- 2026Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy
- 2024AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets
- 2025Large Language Models Achieve Gold Medal Performance at the International Olympiad on Astronomy & Astrophysics (IOAA)
- 2025Reasoning With a Star: A Heliophysics Dataset and Benchmark for Agentic Scientific Reasoning
- 2025OmniScience: A Domain-Specialized LLM for Scientific Reasoning and Discovery
- 2026A systematic evaluation of vision-language models for observational astronomical reasoning tasks