# Fine-Grained Instance-Level Sketch-Based Video Retrieval

arXiv: https://arxiv.org/abs/2002.09461 · HTML: https://mlsanity.com/paper/2002.09461/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2020.

## Built on by (most cited first, 6 of 6)

- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Image-to-Video Generation via 3D Facial Dynamics](https://mlsanity.com/paper/2105.14678/) (2021)
- [Aligning Step-by-Step Instructional Diagrams to Video Demonstrations](https://mlsanity.com/paper/2303.13800/) (2023)
- [Query-guided Attention in Vision Transformers for Localizing Objects Using a Single Sketch](https://mlsanity.com/paper/2303.08784/) (2023)
- [SSR-GNNs: Stroke-based Sketch Representation with Graph Neural Networks](https://mlsanity.com/paper/2204.13153/) (2022)
- [Sketch-based Video Object Localization](https://mlsanity.com/paper/2304.00450/) (2023)

## Abstract (arXiv)

Existing sketch-analysis work studies sketches depicting static objects or scenes. In this work, we propose a novel cross-modal retrieval problem of fine-grained instance-level sketch-based video retrieval (FG-SBVR), where a sketch sequence is used as a query to retrieve a specific target video instance. Compared with sketch-based still image retrieval, and coarse-grained category-level video retrieval, this is more challenging as both visual appearance and motion need to be simultaneously matched at a fine-grained level. We contribute the first FG-SBVR dataset with rich annotations. We then introduce a novel multi-stream multi-modality deep network to perform FG-SBVR under both strong and weakly supervised settings. The key component of the network is a relation module, designed to prevent model over-fitting given scarce training data. We show that this model significantly outperforms a number of existing state-of-the-art models designed for video analysis.