# MDGYM: Benchmarking AI Agents on Molecular Simulations

arXiv: https://arxiv.org/abs/2605.08941 · HTML: https://mlsanity.com/paper/2605.08941/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 9 May 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 22 of 22)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search](https://mlsanity.com/paper/2504.08066/) (2025)
- [SciCode: A Research Coding Benchmark Curated by Scientists](https://mlsanity.com/paper/2407.13168/) (2024)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology](https://mlsanity.com/paper/2503.00096/) (2025)
- [SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories](https://mlsanity.com/paper/2409.07440/) (2024)
- [SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers](https://mlsanity.com/paper/2504.00255/) (2025)
- [SciToolAgent: A Knowledge Graph-Driven Scientific Agent for Multi-Tool Integration](https://mlsanity.com/paper/2507.20280/) (2025)
- [MDCrow: Automating Molecular Dynamics Workflows with Large Language Models](https://mlsanity.com/paper/2502.09565/) (2025)
- [HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks](https://mlsanity.com/paper/2410.12381/) (2024)
- [AI scientists produce results without reasoning scientifically](https://mlsanity.com/paper/2604.18805/) (2026)
- [Automating MD simulations for Proteins using Large language Models: NAMD-Agent](https://mlsanity.com/paper/2507.07887/) (2025)
- [DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle](https://mlsanity.com/paper/2601.20882/) (2026)
- [MDAgent2: Large Language Model for Code Generation and Knowledge Q&A in Molecular Dynamics](https://mlsanity.com/paper/2601.02075/) (2026)
- [DynaMate: An Autonomous Agent for Protein-Ligand Molecular Dynamics Simulations](https://mlsanity.com/paper/2512.10034/) (2025)
- [AInsteinBench: Benchmarking Coding Agents on Scientific Repositories](https://mlsanity.com/paper/2512.21373/) (2025)
- [PolyJarvis: An LLM-Orchestrated Agent for Automated All-Atom Molecular Dynamics of Amorphous Homopolymers](https://mlsanity.com/paper/2604.02537/) (2026)

## Built on by (most cited first, 4 of 4)

- [MDArena: Evaluating Coding Agents on Realistic Molecular Dynamics Workflows](https://mlsanity.com/paper/2608.02642/) (2026)
- [Molecular LLM Agents: From Architectural Design to Scientific Autonomy](https://mlsanity.com/paper/2608.23104/) (2026)
- [CompMat-Bench: Benchmarking AI Agents for Computational Materials Science](https://mlsanity.com/paper/2610.00636/) (2026)
- [NeutronGym: Physics-Graded Neutron Instrument Design for LLM Agents](https://mlsanity.com/paper/2610.03631/) (2026)

## Abstract (arXiv)

The promise of AI-driven scientific discovery hinges on whether AI agents can autonomously design and execute the computational workflows that underpin modern science. Molecular dynamics (MD) simulation presents a natural test bed to stress-test this claim; it requires translating physical intuition into syntactically and semantically correct input scripts, reasoning about initial and boundary conditions, diagnosing numerically unstable trajectories, and interpreting outputs against known physical behavior and laws. We introduce MDGYM, a benchmark of 169 expert-curated MD simulations spanning LAMMPS and GROMACS, two widely used MD packages, across three increasing difficulty levels. We evaluate three agentic frameworks -- Claude Code, Codex, and OpenHands -- with four LLMs, and find that all perform poorly: even the strongest agent solves only 21% of easy-level tasks, with less than 10% at higher difficulties. Trajectory analysis reveals a characteristic pattern of failure -- agents successfully invoke simulation machinery but produce physically unstable configurations, fabricate numerical outputs without executing the underlying computation, or abandon tasks prematurely rather than iterating through simulation-specific errors. These failure modes are qualitatively distinct from those observed in general software engineering benchmarks, indicating that fluent code generation does not transfer to grounded physical reasoning.