Grounded autonomous scrutiny at scale: emergent critique from reproduction of published computational physics papers
On the map: LLM agents and tool use → Web, search, and research agents
Abstract (arXiv)
Autonomous LLM agents now produce complete research artifacts in machine-learning sandboxes, but real computational physics is harder: experiments are first-principles calculations against re-runnable physical ground truth, and meaningful new work almost always builds on a key existing paper. We ask whether such an agent can perform grounded scrutiny of published computational physics - reading a paper, reproducing it from scratch, and surfacing methodological concerns from execution. We deploy a single Claude Opus 4.6 configuration at two complementary scopes. At scale, across 111 open-access Quantum ESPRESSO papers, an autonomous agent runs the read-plan-compute-compare loop and, although never asked to critique, raises substantive methodological concerns on ~42% of papers; 85 of 88 of these critiques (96.6%) surface only after the agent has actually run a calculation, with a reading-only ceiling of 1.8%. Critique emerges from reproduction, not from reading. In depth, on one Nature Communications paper on multiscale device simulation of a 2D-material MOSFET, a fresh agent inheriting a verified reproduction pipeline autonomously produces a 14-concern physics inventory and a complete, submission-form six-page Comment that revises the paper's L_G = 5 nm headline. Two of its L_G = 5 nm headline-challenging attacks - a source-degeneration contact-resistance bound and a Sb-doping degradation ratio - are absent from the published 21-reviewer peer review.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 9 of its 36 references that are in the map. First posted on arXiv 14 Apr 2026.
Builds on
9 of 36 references.
Closest ancestors · 1
Same direction, newest first
Other references · 8
- 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
- 2022OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and conceptscited by 166
- 2026Towards End-to-End Automation of AI Researchcited by 150
- 2025El Agente: An Autonomous Agent for Quantum Chemistrycited by 43
- 2025Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?cited by 32
- 2025DREAMS: Density Functional Theory Based Research Engine for Agentic Materials Simulationcited by 29
- 2025When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchcited by 25
- 2026From Experiments to Expertise: Scientific Knowledge Consolidation for AI-Driven Computational Physicscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Grounded autonomous research: a fault-tolerant LLM pipeline from corpus to manuscript in frontier computational physics
- 2025PhysMaster: Building an Autonomous AI Physicist for Theoretical and Computational Physics Research
- 2026Evaluating a 4B open-weights local LLM for agentic DFT workflows: a literature reproducibility audit
- 2026AI Agents Can Already Autonomously Perform Experimental High Energy Physics
- 2026PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research
- 2025GENIUS: An Agentic AI Framework for Autonomous Design and Execution of Simulation Protocols
- 2025Can Theoretical Physics Research Benefit from Language Agents?
- 2025Towards autonomous quantum physics research using LLM agents with access to intelligent tools