mlsanityData to 7 Oct 2026

PHITSBench: an execution-scored benchmark for AI-assisted PHITS radiation-transport input generation using natural language

arXiv 2607.09789 ↗First posted 8 Jul 2026cs.AI cond-mat.mtrl-scicited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce PHITSBench, an execution-scored benchmark for the Monte Carlo Particle and Heavy Ion Transport code System (PHITS). PHITSBench comprises 282 transport-scorable tasks spanning three common workflow categories: parameter editing (Edit), syntax repair (Repair ), and complete simulation generation from natural-language descriptions (Reproduce). Each task is evaluated using a Composite Metric Score that combines execution success with agreement between generated and reference transport observables. Using PHITSBench, we evaluate five GPT-5.4-based configurations ranging from zero-shot prompting to knowledge-augmented and agentic workflows. Without domain-specific knowledge, the model performs well on editing and repair tasks (95% and 70% success, respectively) but fails to generate correct simulations from scratch (0% success on the Reproduce track). A structured, machine-readable PHITS knowledge catalog, supplied alongside the user manual, raises single-shot Reproduce-task success to 57%. Agentic execution provides a further improvement to 66-73%, but at increased computational cost. Failure analysis shows that the remaining errors are dominated by incorrect selection and configuration of physical observables rather than syntax generation. These results suggest that future progress in AI-assisted radiation-transport modeling will depend as much on machine-readable knowledge bases, curated domain-training datasets, and execution-grounded evaluation environments as on advances in foundation models themselves.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 48 references that are in the map. First posted on arXiv 8 Jul 2026.

Builds on

24 of 48 references.

References · 24

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  6. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  7. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  8. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  9. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  10. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  11. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  12. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 246
12 more
  1. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  2. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 191
  3. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  4. 2022DocPrompting: Generating Code by Retrieving the Docscited by 151
  5. 2024MetaOpenFOAM: an LLM-based multi-agent framework for CFDcited by 39
  6. 2025MooseAgent: A LLM Based Multi-agent Framework for Automating Moose Simulationcited by 16
  7. 2024DocCGen: Document-based Controlled Code Generationcited by 11
  8. 2026GRACE: an Agentic AI for Particle Physics Experiment Design and Simulationcited by 5
  9. 2025Exploring the Capabilities of the Frontier Large Language Models for Nuclear Energy Researchcited by 2
  10. 2025SIMCODE: A Benchmark for Natural Language to ns-3 Network Simulation Code Generationcited by 2
  11. 2026Evaluating LLM-generated code for domain-specific languages: molecular dynamics with LAMMPScited by 2
  12. 2026AutoSAM: an Agentic Framework for Automating Input File Generation for the SAM Code with Multi-Modal Retrieval-Augmented Generationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reliable LLM-Generated Programs for High-Energy Physics Experiments through Graph-Grounded Software Knowledge ↗
  2. 2025Agentic Artificial Intelligence for Multistage Physics Experiments at a Large-Scale User Facility Particle Accelerator
  3. 2025CelloAI: Leveraging Large Language Models for HPC Software Development in High Energy Physics
  4. 2026PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinement
  5. 2025EnvTrace: Simulation-Based Semantic Evaluation of LLM Code via Execution Trace Alignment -- Demonstrated at Synchrotron Beamlines
  6. 2026AI Agents Can Already Autonomously Perform Experimental High Energy Physics
  7. 2025Generative Monte Carlo Sampling for Constant-Cost Particle Transport
  8. 2025Chain of Unit-Physics: A Primitive-Centric Approach to Scientific Code Synthesis