# LabInstruct: Benchmarking Situated Instructional Video Generation for Lab Procedures

arXiv: https://arxiv.org/abs/2610.14020 · HTML: https://mlsanity.com/paper/2610.14020/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 47 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 29 of 29)

- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Accelerating scientific discovery with Co-Scientist](https://mlsanity.com/paper/2502.18864/) (2025)
- [VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness](https://mlsanity.com/paper/2503.21755/) (2025)
- [LTX-2: Efficient Joint Audio-Visual Foundation Model](https://mlsanity.com/paper/2601.03233/) (2026)
- [Do generative video models understand physical principles?](https://mlsanity.com/paper/2501.09038/) (2025)
- [VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation](https://mlsanity.com/paper/2503.06800/) (2025)
- [WorldModelBench: Judging Video Generation Models As World Models](https://mlsanity.com/paper/2502.20694/) (2025)
- [An AI system to help scientists write expert-level empirical software](https://mlsanity.com/paper/2509.06503/) (2025)
- [Seer: Language Instructed Video Prediction with Latent Diffusion Models](https://mlsanity.com/paper/2303.14897/) (2023)
- [ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows](https://mlsanity.com/paper/2505.19897/) (2025)
- [Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems](https://mlsanity.com/paper/2504.02918/) (2025)
- [GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos](https://mlsanity.com/paper/2312.07322/) (2023)
- [ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Maps](https://mlsanity.com/paper/2505.18675/) (2025)
- [AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction](https://mlsanity.com/paper/2406.06465/) (2024)
- [RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning](https://mlsanity.com/paper/2510.02240/) (2025)
- [AutoBio: A Simulation and Benchmark for Robotic Automation in Digital Biology Laboratory](https://mlsanity.com/paper/2505.14030/) (2025)
- [FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotation](https://mlsanity.com/paper/2402.00293/) (2024)
- [ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions](https://mlsanity.com/paper/2412.01987/) (2024)
- [Benchmarking Scientific Understanding and Reasoning for Video Generation using VideoScience-Bench](https://mlsanity.com/paper/2512.02942/) (2025)
- [ExpVid: A Benchmark for Experiment Video Understanding & Reasoning](https://mlsanity.com/paper/2510.11606/) (2025)
- [PhyEduVideo: A Benchmark for Evaluating Text-to-Video Models for Physics Education](https://mlsanity.com/paper/2601.00943/) (2026)
- [PhysLab: A Benchmark Dataset for Multi-Granularity Visual Parsing of Physics Experiments](https://mlsanity.com/paper/2506.06631/) (2025)
- [Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistance](https://mlsanity.com/paper/2512.13238/) (2025)
- [Show Me: Unifying Instructional Image and Video Generation with Diffusion Models](https://mlsanity.com/paper/2511.17839/) (2025)
- [How to Correctly Make Mistakes: A Framework for Constructing and Benchmarking Mistake Aware Egocentric Procedural Videos](https://mlsanity.com/paper/2604.15134/) (2026)
- [Demo2Tutorial: From Human Experience to Multimodal Software Tutorials](https://mlsanity.com/paper/2606.03951/) (2026)
- [Sci-VBench: Evaluating Knowledge- and Reasoning-Intensive Video Generation in Science Domains](https://mlsanity.com/paper/2608.09873/) (2026)
- [Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding](https://mlsanity.com/paper/2605.15342/) (2026)

## Abstract (arXiv)

Self-driving laboratories (SDLs) aim to automate the full experimental loop, from scientific decision-making to physical execution. Ideally, AI-generated plans could be carried out directly by robotic systems, but reliable automation remains difficult in complex, open-world laboratory environments, where experiments often involve fine-grained manipulation, long-horizon procedures, and substantial variation across tasks and setups. Humans therefore remain an important execution interface between AI-generated plans and physical experiments, creating a need for clear and effective human-facing experimental guidance. Because laboratory procedures are inherently visual, spatial, and dynamic, video is particularly well suited to communicating apparatus configurations, manipulation actions, temporal dependencies, and state changes. Recent advances in video generation now make it possible to synthesize experimental demonstrations directly from an initial workspace image and a natural-language instruction. However, whether such models can reliably communicate real laboratory procedures has not been systematically studied. We introduce LabInstruct, a benchmark for situated instructional video generation in real laboratories. LabInstruct contains 204 tasks across 5 scientific disciplines, with real reference executions and structured annotations of objects, actions, contacts, and state transitions. Evaluating 8 frontier image-to-video models, we find that visually plausible generations frequently remain procedurally incorrect, revealing a substantial gap between visual realism and the reliability required for experimental instruction.