# onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

arXiv: https://arxiv.org/abs/2608.02595 · HTML: https://mlsanity.com/paper/2608.02595/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 34 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Builds on (most cited first, 8 of 8)

- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks](https://mlsanity.com/paper/2305.18365/) (2023)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [Towards End-to-End Automation of AI Research](https://mlsanity.com/paper/2606.15497/) (2026)
- [ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domain](https://mlsanity.com/paper/2411.16736/) (2024)
- [Assessing the Chemical Intelligence of Large Language Models](https://mlsanity.com/paper/2505.07735/) (2025)
- [SMILES-Prompting: A Novel Approach to LLM Jailbreak Attacks in Chemical Synthesis](https://mlsanity.com/paper/2410.15641/) (2024)
- [Evaluating the Progression of Large Language Model Capabilities for Small-Molecule Drug Design](https://mlsanity.com/paper/2604.16279/) (2026)

## Abstract (arXiv)

Language models are playing an increasingly important role in laboratory science, performing tasks such as experiment planning, execution, and post-hoc analysis. However, precisely measuring their abilities is difficult, as scientific capabilities require a mixture of both problem-solving skills and domain-specific intuition. Existing evaluations rarely measure the capabilities required to make reliable decisions in a physical laboratory and often rely on public data that may have appeared in model training corpora. We introduce onepot-Bench 0, a proprietary benchmark suite for evaluating language models on synthetic chemistry capabilities relevant to wet-lab execution. onepot-Bench 0 comprises three complementary evaluations: ChemAbacus measures tool-free cheminformatics literacy and numerical reasoning; SynthRefusal characterizes safety and refusal behavior across a variety of benign, controlled, and designer-drug targets; and SynthBench evaluates reaction-outcome prediction and catalyst selection using private experimental data generated in our laboratory. Together, these evaluations probe basic competency, reliability, and deeper knowledge, all skills which are required for reliable performance in the lab.