# A Dataset for Medical Instructional Video Classification and Question Answering

arXiv: https://arxiv.org/abs/2201.12888 · HTML: https://mlsanity.com/paper/2201.12888/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 30 Jan 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation](https://mlsanity.com/paper/2002.06353/) (2020)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression](https://mlsanity.com/paper/1804.07014/) (2018)
- [TVQA+: Spatio-Temporal Grounding for Video Question Answering](https://mlsanity.com/paper/1904.11574/) (2019)
- [Localizing Moments in Video with Temporal Language](https://mlsanity.com/paper/1809.01337/) (2018)
- [MarioQA: Answering Questions by Watching Gameplay Videos](https://mlsanity.com/paper/1612.01669/) (2016)
- [Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detection](https://mlsanity.com/paper/1809.05884/) (2018)

## Built on by (most cited first, 26 of 26)

- [MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models](https://mlsanity.com/paper/2501.02955/) (2025)
- [Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models](https://mlsanity.com/paper/2504.15271/) (2025)
- [Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Video](https://mlsanity.com/paper/2203.06667/) (2022)
- [FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment](https://mlsanity.com/paper/2405.06887/) (2024)
- [NurViD: A Large Expert-Level Video Database for Nursing Procedure Activity Understanding](https://mlsanity.com/paper/2310.13347/) (2023)
- [Large Language Model Benchmarks in Medical Tasks](https://mlsanity.com/paper/2410.21348/) (2024)
- [Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositions](https://mlsanity.com/paper/2504.11373/) (2025)
- [Learning to Locate Visual Answer in Video Corpus Using Question](https://mlsanity.com/paper/2210.05423/) (2022)
- [Visual Answer Localization with Cross-modal Mutual Knowledge Transfer](https://mlsanity.com/paper/2210.14823/) (2022)
- [Can Vision-Language Models Answer Face to Face Questions in the Real-World?](https://mlsanity.com/paper/2503.19356/) (2025)
- [How Well Can General Vision-Language Models Learn Medicine By Watching Public Educational Videos?](https://mlsanity.com/paper/2504.14391/) (2025)
- [From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine](https://mlsanity.com/paper/2502.09242/) (2025)
- [Medical Image Retrieval via Nearest Neighbor Search on Pre-trained Image Features](https://mlsanity.com/paper/2210.02401/) (2022)
- [EyePCR: A Comprehensive Benchmark for Fine-Grained Perception, Knowledge Comprehension and Clinical Reasoning in Ophthalmic Surgery](https://mlsanity.com/paper/2509.15596/) (2025)
- [Human in the loop approaches in multi-modal conversational task guidance system development](https://mlsanity.com/paper/2211.01824/) (2022)
- [TVR-Ranking: A Dataset for Ranked Video Moment Retrieval with Imprecise Queries](https://mlsanity.com/paper/2407.06597/) (2024)
- [Overview of TREC 2024 Medical Video Question Answering (MedVidQA) Track](https://mlsanity.com/paper/2412.11056/) (2024)
- [Ask2Loc: Learning to Locate Instructional Visual Answers by Asking Questions](https://mlsanity.com/paper/2504.15918/) (2025)
- [Med-CRAFT: An Information System for Explainable and Configurable Construction of Multimodal Medical QA Datasets](https://mlsanity.com/paper/2512.01045/) (2025)
- [Towards Answering Health-related Questions from Medical Videos: Datasets and Approaches](https://mlsanity.com/paper/2309.12224/) (2023)
- [Enhancing the Learning Experience: Using Vision-Language Models to Generate Questions for Educational Videos](https://mlsanity.com/paper/2505.01790/) (2025)
- [Overview of the NLPCC 2026 Shared Task 1: Difficulty-Aware Multilingual and Multimodal Medical Instructional Video Understanding Evaluation](https://mlsanity.com/paper/2607.06618/) (2026)
- [Multimodal Transformer With a Low-Computational-Cost Guarantee](https://mlsanity.com/paper/2402.15096/) (2024)
- [Seizure-Semiology-Suite (S3): A Clinically Multimodal Dataset, Benchmark, and Models for Seizure Semiology Understanding](https://mlsanity.com/paper/2605.21852/) (2026)
- [DAEP: Difficulty-Aware Evidence Planning for Medical Video Corpus Temporal Answer Grounding](https://mlsanity.com/paper/2608.06869/) (2026)
- [DynamicDx: Evaluating Evidence Acquisition in Video-Based Diagnosis](https://mlsanity.com/paper/2609.32957/) (2026)

## Abstract (arXiv)

This paper introduces a new challenge and datasets to foster research toward designing systems that can understand medical videos and provide visual answers to natural language questions. We believe medical videos may provide the best possible answers to many first aids, medical emergency, and medical education questions. Toward this, we created the MedVidCL and MedVidQA datasets and introduce the tasks of Medical Video Classification (MVC) and Medical Visual Answer Localization (MVAL), two tasks that focus on cross-modal (medical language and medical video) understanding. The proposed tasks and datasets have the potential to support the development of sophisticated downstream applications that can benefit the public and medical practitioners. Our datasets consist of 6,117 annotated videos for the MVC task and 3,010 annotated questions and answers timestamps from 899 videos for the MVAL task. These datasets have been verified and corrected by medical informatics experts. We have also benchmarked each task with the created MedVidCL and MedVidQA datasets and proposed the multimodal learning methods that set competitive baselines for future research.