# Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraints

arXiv: https://arxiv.org/abs/2604.15664 · HTML: https://mlsanity.com/paper/2604.15664/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 51 references that are in the map.
- First posted on arXiv 17 Apr 2026.

## Builds on (most cited first, 17 of 17)

- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [SciCode: A Research Coding Benchmark Curated by Scientists](https://mlsanity.com/paper/2407.13168/) (2024)
- [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024)
- [On scientific understanding with artificial intelligence](https://mlsanity.com/paper/2204.01467/) (2022)
- [UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models](https://mlsanity.com/paper/2502.00334/) (2025)
- [SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoning](https://mlsanity.com/paper/2505.19099/) (2025)
- [Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark](https://mlsanity.com/paper/2509.26574/) (2025)
- [Gravity-Bench-v1: A Benchmark on Gravitational Physics Discovery for Agents](https://mlsanity.com/paper/2501.18411/) (2025)
- [AstroVisBench: A Code Benchmark for Scientific Computing and Visualization in Astronomy](https://mlsanity.com/paper/2505.20538/) (2025)
- [Solving an Open Problem in Theoretical Physics using AI-Assisted Discovery](https://mlsanity.com/paper/2603.04735/) (2026)
- [PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning](https://mlsanity.com/paper/2510.03185/) (2025)
- [AstroMMBench: A Benchmark for Evaluating Multimodal Large Language Models Capabilities in Astronomy](https://mlsanity.com/paper/2510.00063/) (2025)
- [Probabilistic Cross-Identification in Crowded Fields as an Assignment Problem](https://mlsanity.com/paper/1609.03065/) (2016)
- [AstroReason-Bench: Evaluating Unified Agentic Planning across Heterogeneous Space Planning Problems](https://mlsanity.com/paper/2601.11354/) (2026)

## Built on by (most cited first, 3 of 3)

- [Can LLMs Discover Scientific Laws in Real and Parallel Worlds?](https://mlsanity.com/paper/2609.01552/) (2026)
- [NeutronGym: Physics-Graded Neutron Instrument Design for LLM Agents](https://mlsanity.com/paper/2610.03631/) (2026)
- [LLM-IDEA: Identifiability-Driven Experimental Agent for Autonomous Discovery of Mechanistic World Models](https://mlsanity.com/paper/2610.11253/) (2026)

## Abstract (arXiv)

The rise of autonomous AI agents suggests that dynamic benchmark environments with built-in feedback on scientifically grounded tasks are needed to evaluate the capabilities of these agents in research work. We introduce Stargazer, a scalable environment for evaluating AI agents on dynamic, iterative physics-grounded model-fitting tasks using inference on radial-velocity (RV) time series data. Stargazer comprises 120 tasks across three difficulty tiers, including 20 real archival cases, covering diverse scenarios ranging from high-SNR single-planet systems to complex multi-planetary configurations requiring involved low-SNR analysis. Our evaluation of eight frontier agents reveals a gap between numerical optimization and adherence to physical constraints: although agents often achieve a good statistical fit, they frequently fail to recover correct physical system parameters, a limitation that persists even when agents are equipped with vanilla skills. Furthermore, increasing test-time compute yields only marginal gains, with excessive token usage often reflecting recursive failure loops rather than meaningful exploration. Stargazer presents an opportunity to train, evaluate, scaffold, and scale strategies on a model-fitting problem of practical research relevance today. Our methodology to design a simulation-driven environment for AI agents presumably generalizes to many other model-fitting problems across scientific domains. Source code and the project website are available at https://github.com/AIPS-UofT/Stargazer and https://aips-uoft.github.io/Stargazer/, respectively.