# Visual Answer Localization with Cross-modal Mutual Knowledge Transfer

arXiv: https://arxiv.org/abs/2210.14823 · HTML: https://mlsanity.com/paper/2210.14823/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 13 of 13)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [WikiHow: A Large Scale Text Summarization Dataset](https://mlsanity.com/paper/1810.09305/) (2018)
- [QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension](https://mlsanity.com/paper/1804.09541/) (2018)
- [To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression](https://mlsanity.com/paper/1804.07014/) (2018)
- [Proposal-free Temporal Moment Localization of a Natural-Language Query in Video using Guided Attention](https://mlsanity.com/paper/1908.07236/) (2019)
- [Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Video](https://mlsanity.com/paper/2203.06667/) (2022)
- [Relation-aware Video Reading Comprehension for Temporal Language Grounding](https://mlsanity.com/paper/2110.05717/) (2021)
- [Frame-wise Cross-modal Matching for Video Moment Retrieval](https://mlsanity.com/paper/2009.10434/) (2020)
- [A Dataset for Medical Instructional Video Classification and Question Answering](https://mlsanity.com/paper/2201.12888/) (2022)
- [TutorialVQA: Question Answering Dataset for Tutorial Videos](https://mlsanity.com/paper/1912.01046/) (2019)

## Built on by (most cited first, 7 of 7)

- [Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networks](https://mlsanity.com/paper/2304.01665/) (2023)
- [Overview of the NLPCC 2025 Shared Task 4: Multi-modal, Multilingual, and Multi-hop Medical Instructional Video Question Answering Challenge](https://mlsanity.com/paper/2505.06814/) (2025)
- [Learning to Unify Audio, Visual and Text for Audio-Enhanced Multilingual Visual Answer Localization](https://mlsanity.com/paper/2411.02851/) (2024)
- [Ask2Loc: Learning to Locate Instructional Visual Answers by Asking Questions](https://mlsanity.com/paper/2504.15918/) (2025)
- [Hierarchical Indexing with Knowledge Enrichment for Multilingual Video Corpus Retrieval](https://mlsanity.com/paper/2510.09553/) (2025)
- [Overview of the NLPCC 2026 Shared Task 1: Difficulty-Aware Multilingual and Multimodal Medical Instructional Video Understanding Evaluation](https://mlsanity.com/paper/2607.06618/) (2026)
- [DAEP: Difficulty-Aware Evidence Planning for Medical Video Corpus Temporal Answer Grounding](https://mlsanity.com/paper/2608.06869/) (2026)

## Abstract (arXiv)

The goal of visual answering localization (VAL) in the video is to obtain a relevant and concise time clip from a video as the answer to the given natural language question. Early methods are based on the interaction modelling between video and text to predict the visual answer by the visual predictor. Later, using the textual predictor with subtitles for the VAL proves to be more precise. However, these existing methods still have cross-modal knowledge deviations from visual frames or textual subtitles. In this paper, we propose a cross-modal mutual knowledge transfer span localization (MutualSL) method to reduce the knowledge deviation. MutualSL has both visual predictor and textual predictor, where we expect the prediction results of these both to be consistent, so as to promote semantic knowledge understanding between cross-modalities. On this basis, we design a one-way dynamic loss function to dynamically adjust the proportion of knowledge transfer. We have conducted extensive experiments on three public datasets for evaluation. The experimental results show that our method outperforms other competitive state-of-the-art (SOTA) methods, demonstrating its effectiveness.