# Learning to Locate Visual Answer in Video Corpus Using Question

arXiv: https://arxiv.org/abs/2210.05423 · HTML: https://mlsanity.com/paper/2210.05423/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 16 of 16)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [WikiHow: A Large Scale Text Summarization Dataset](https://mlsanity.com/paper/1810.09305/) (2018)
- [RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking](https://mlsanity.com/paper/2110.07367/) (2021)
- [TVQA+: Spatio-Temporal Grounding for Video Question Answering](https://mlsanity.com/paper/1904.11574/) (2019)
- [MDMMT: Multidomain Multimodal Transformer for Video Retrieval](https://mlsanity.com/paper/2103.10699/) (2021)
- [Video Corpus Moment Retrieval with Contrastive Learning](https://mlsanity.com/paper/2105.06247/) (2021)
- [Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Video](https://mlsanity.com/paper/2203.06667/) (2022)
- [A Survey on Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2109.08039/) (2021)
- [Frame-wise Cross-modal Matching for Video Moment Retrieval](https://mlsanity.com/paper/2009.10434/) (2020)
- [A Dataset for Medical Instructional Video Classification and Question Answering](https://mlsanity.com/paper/2201.12888/) (2022)

## Built on by (most cited first, 10 of 10)

- [Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networks](https://mlsanity.com/paper/2304.01665/) (2023)
- [Overview of the NLPCC 2025 Shared Task 4: Multi-modal, Multilingual, and Multi-hop Medical Instructional Video Question Answering Challenge](https://mlsanity.com/paper/2505.06814/) (2025)
- [Learning to Unify Audio, Visual and Text for Audio-Enhanced Multilingual Visual Answer Localization](https://mlsanity.com/paper/2411.02851/) (2024)
- [OSEF: One-Step Evidence Fusion for Cross-Video Scene Procedure Planning](https://mlsanity.com/paper/2607.29401/) (2026)
- [Ask2Loc: Learning to Locate Instructional Visual Answers by Asking Questions](https://mlsanity.com/paper/2504.15918/) (2025)
- [M3-Med: A Benchmark for Multi-lingual, Multi-modal, and Multi-hop Reasoning in Medical Instructional Video Understanding](https://mlsanity.com/paper/2507.04289/) (2025)
- [Towards Answering Health-related Questions from Medical Videos: Datasets and Approaches](https://mlsanity.com/paper/2309.12224/) (2023)
- [Hierarchical Indexing with Knowledge Enrichment for Multilingual Video Corpus Retrieval](https://mlsanity.com/paper/2510.09553/) (2025)
- [Overview of the NLPCC 2026 Shared Task 1: Difficulty-Aware Multilingual and Multimodal Medical Instructional Video Understanding Evaluation](https://mlsanity.com/paper/2607.06618/) (2026)
- [DAEP: Difficulty-Aware Evidence Planning for Medical Video Corpus Temporal Answer Grounding](https://mlsanity.com/paper/2608.06869/) (2026)

## Abstract (arXiv)

We introduce a new task, named video corpus visual answer localization (VCVAL), which aims to locate the visual answer in a large collection of untrimmed instructional videos using a natural language question. This task requires a range of skills - the interaction between vision and language, video retrieval, passage comprehension, and visual answer localization. In this paper, we propose a cross-modal contrastive global-span (CCGS) method for the VCVAL, jointly training the video corpus retrieval and visual answer localization subtasks with the global-span matrix. We have reconstructed a dataset named MedVidCQA, on which the VCVAL task is benchmarked. Experimental results show that the proposed method outperforms other competitive methods both in the video corpus retrieval and visual answer localization subtasks. Most importantly, we perform detailed analyses on extensive experiments, paving a new path for understanding the instructional videos, which ushers in further research.