# LabOPBench: A Multimodal Benchmark for Evidence-Grounded Laboratory Decision Making

arXiv: https://arxiv.org/abs/2610.19780 · HTML: https://mlsanity.com/paper/2610.19780/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 14 Oct 2026.

## Builds on (most cited first, 14 of 14)

- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks](https://mlsanity.com/paper/2305.18365/) (2023)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area](https://mlsanity.com/paper/2408.07246/) (2024)
- [ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterization](https://mlsanity.com/paper/2401.06949/) (2024)
- [LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labs](https://mlsanity.com/paper/2410.14182/) (2024)
- [MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension](https://mlsanity.com/paper/2403.08192/) (2024)
- [Benchmarking Multimodal LLMs on Recognition and Understanding over Chemical Tables](https://mlsanity.com/paper/2506.11375/) (2025)
- [Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on Materials Characterization](https://mlsanity.com/paper/2509.09307/) (2025)
- [QCBench: Evaluating Large Language Models on Domain-Specific Quantitative Chemistry](https://mlsanity.com/paper/2508.01670/) (2025)
- [Evaluating Large Language Models on Multimodal Chemistry Olympiad Exams](https://mlsanity.com/paper/2512.14989/) (2025)
- [A VLM-based Method for Visual Anomaly Detection in Robotic Scientific Laboratories](https://mlsanity.com/paper/2506.05405/) (2025)
- [RxnBench: A Multimodal Benchmark for Evaluating Large Language Models on Chemical Reaction Understanding from Scientific Literature](https://mlsanity.com/paper/2512.23565/) (2025)

## Abstract (arXiv)

Laboratory experiments involve more than following predefined protocols; they require adaptive decision-making as the experimental state changes. Existing scientific benchmarks rarely assess such evidence-based decisions in laboratory settings. Here we present LabOPBench, a multimodal benchmark of 756 expert-authored, independently reviewed questions in organic synthesis and materials science. Each item pairs laboratory evidence with procedural context and an open-ended judgment, evaluated using a question-specific scoring rubric. Eight multimodal models obtain overall normalized scores of 28.8-45.8%, leaving substantial room for improvement. On a controlled 100-question subset, the strongest models outperform undergraduate participants on average but remain below the average performance of PhD participants. Expert analysis of responses from the three leading models identifies Evidence Misreading and Evidence-Decision Conflict as the dominant error patterns. These results reveal that current models still struggle to turn multimodal perception into reliable laboratory decision-making.