# InstructionBench: An Instructional Video Understanding Benchmark

arXiv: https://arxiv.org/abs/2504.05040 · HTML: https://mlsanity.com/paper/2504.05040/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Builds on (most cited first, 17 of 17)

- [CLOTH3D: Clothed 3D Humans](https://mlsanity.com/paper/1912.02792/) (2019)
- [Learning Predictive Representations for Deformable Objects Using Contrastive Estimation](https://mlsanity.com/paper/2003.05436/) (2020)
- [GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completion](https://mlsanity.com/paper/2104.05177/) (2021)
- [Challenges and Outlook in Robotic Manipulation of Deformable Objects](https://mlsanity.com/paper/2105.01767/) (2021)
- [VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation](https://mlsanity.com/paper/2003.09044/) (2020)
- [Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation](https://mlsanity.com/paper/2206.02881/) (2022)
- [Folding Deformable Objects using Predictive Simulation and Trajectory Optimization](https://mlsanity.com/paper/1512.06922/) (2015)
- [Unfolding the Literature: A Review of Robotic Cloth Manipulation](https://mlsanity.com/paper/2407.01361/) (2024)
- [Cloth Region Segmentation for Robust Grasp Selection](https://mlsanity.com/paper/2008.05626/) (2020)
- [GarmentTracking: Category-Level Garment Pose Tracking](https://mlsanity.com/paper/2303.13913/) (2023)
- [Learning to Reconstruct Texture-less Deformable Surfaces from a Single View](https://mlsanity.com/paper/1803.08908/) (2018)
- [TRTM: Template-based Reconstruction and Target-oriented Manipulation of Crumpled Cloths](https://mlsanity.com/paper/2308.04670/) (2023)
- [Reinforcement Learning in Topology-based Representation for Human Body Movement with Whole Arm Manipulation](https://mlsanity.com/paper/1809.04322/) (2018)
- [Enabling Visual Action Planning for Object Manipulation through Latent Space Roadmap](https://mlsanity.com/paper/2103.02554/) (2021)
- [Grasp-Oriented Fine-grained Cloth Segmentation without Real Supervision](https://mlsanity.com/paper/2110.02903/) (2021)
- [BiFold: Bimanual Cloth Folding with Language Guidance](https://mlsanity.com/paper/2501.16458/) (2025)
- [A novel collision model for inextensible textiles and its experimental validation](https://mlsanity.com/paper/2303.14437/) (2023)

## Built on by (most cited first, 5 of 5)

- [MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks](https://mlsanity.com/paper/2506.07016/) (2025)
- [CausalStep: A Benchmark for Explicit Stepwise Causal Reasoning in Videos](https://mlsanity.com/paper/2507.16878/) (2025)
- [AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning](https://mlsanity.com/paper/2508.07470/) (2025)
- [Watch, Remember, Reason: Human-View Video Understanding with MLLMs](https://mlsanity.com/paper/2606.07433/) (2026)
- [I2G: Generating Instructional Illustrations via Text-Conditioned Diffusion](https://mlsanity.com/paper/2505.16425/) (2025)

## Abstract (arXiv)

Despite progress in video large language models (Video-LLMs), research on instructional video understanding, crucial for enhancing access to instructional content, remains insufficient. To address this, we introduce InstructionBench, an Instructional video understanding Benchmark, which challenges models' advanced temporal reasoning within instructional videos characterized by their strict step-by-step flow. Employing GPT-4, we formulate Q&A pairs in open-ended and multiple-choice formats to assess both Coarse-Grained event-level and Fine-Grained object-level reasoning. Our filtering strategies exclude questions answerable purely by common-sense knowledge, focusing on visual perception and analysis when evaluating Video-LLM models. The benchmark finally contains 5k questions across over 700 videos. We evaluate the latest Video-LLMs on our InstructionBench, finding that closed-source models outperform open-source ones. However, even the best model, GPT-4o, achieves only 53.42% accuracy, indicating significant gaps in temporal reasoning. To advance the field, we also develop a comprehensive instructional video dataset with over 19k Q&A pairs from nearly 2.5k videos, using an automated data generation framework, thereby enriching the community's research resources. All data are available at https://huggingface.co/datasets/sunwhw/InstructionBench.