mlsanityData to 7 Oct 2026

Grounded autonomous scrutiny at scale: emergent critique from reproduction of published computational physics papers

arXiv 2604.12198 ↗First posted 14 Apr 2026physics.comp-ph cond-mat.mtrl-sci cs.AIcited by 2

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Autonomous LLM agents now produce complete research artifacts in machine-learning sandboxes, but real computational physics is harder: experiments are first-principles calculations against re-runnable physical ground truth, and meaningful new work almost always builds on a key existing paper. We ask whether such an agent can perform grounded scrutiny of published computational physics - reading a paper, reproducing it from scratch, and surfacing methodological concerns from execution. We deploy a single Claude Opus 4.6 configuration at two complementary scopes. At scale, across 111 open-access Quantum ESPRESSO papers, an autonomous agent runs the read-plan-compute-compare loop and, although never asked to critique, raises substantive methodological concerns on ~42% of papers; 85 of 88 of these critiques (96.6%) surface only after the agent has actually run a calculation, with a reading-only ceiling of 1.8%. Critique emerges from reproduction, not from reading. In depth, on one Nature Communications paper on multiscale device simulation of a 2D-material MOSFET, a fresh agent inheriting a verified reproduction pipeline autonomously produces a 14-concern physics inventory and a complete, submission-form six-page Comment that revises the paper's L_G = 5 nm headline. Two of its L_G = 5 nm headline-challenging attacks - a source-degeneration contact-resistance bound and a Sb-doping degradation ratio - are absent from the published 21-reviewer peer review.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 9 of its 36 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

9 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark17 Sep

Other references · 8

  1. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  2. 2022OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and conceptscited by 166
  3. 2026Towards End-to-End Automation of AI Researchcited by 150
  4. 2025El Agente: An Autonomous Agent for Quantum Chemistrycited by 43
  5. 2025Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?cited by 32
  6. 2025DREAMS: Density Functional Theory Based Research Engine for Agentic Materials Simulationcited by 29
  7. 2025When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchcited by 25
  8. 2026From Experiments to Expertise: Scientific Knowledge Consolidation for AI-Driven Computational Physicscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Grounded autonomous research: a fault-tolerant LLM pipeline from corpus to manuscript in frontier computational physics
  2. 2025PhysMaster: Building an Autonomous AI Physicist for Theoretical and Computational Physics Research
  3. 2026Evaluating a 4B open-weights local LLM for agentic DFT workflows: a literature reproducibility audit
  4. 2026AI Agents Can Already Autonomously Perform Experimental High Energy Physics
  5. 2026PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research
  6. 2025GENIUS: An Agentic AI Framework for Autonomous Design and Execution of Simulation Protocols
  7. 2025Can Theoretical Physics Research Benefit from Language Agents?
  8. 2025Towards autonomous quantum physics research using LLM agents with access to intelligent tools