# ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning

arXiv: https://arxiv.org/abs/1908.07410 · HTML: https://mlsanity.com/paper/1908.07410/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Aug 2019.

## Built on by (most cited first, 30 of 42)

- [Video Transformers: A Survey](https://mlsanity.com/paper/2201.05991/) (2022)
- [The 2021 Image Similarity Dataset and Challenge](https://mlsanity.com/paper/2106.09672/) (2021)
- [Deep Spatially and Temporally Aware Similarity Computation for Road Network Constrained Trajectories](https://mlsanity.com/paper/2112.09339/) (2021)
- [UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2111.14799/) (2021)
- [DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval](https://mlsanity.com/paper/2106.13266/) (2021)
- [TransVCL: Attention-enhanced Video Copy Localization Network with Flexible Supervision](https://mlsanity.com/paper/2211.13090/) (2022)
- [Video Similarity and Alignment Learning on Partial Video Copy Detection](https://mlsanity.com/paper/2108.01817/) (2021)
- [Self-Supervised Video Similarity Learning](https://mlsanity.com/paper/2304.03378/) (2023)
- [Frame by Familiar Frame: Understanding Replication in Video Diffusion Models](https://mlsanity.com/paper/2403.19593/) (2024)
- [A Large-scale Comprehensive Dataset and Copy-overlap Aware Evaluation Protocol for Segment-level Video Copy Detection](https://mlsanity.com/paper/2203.02654/) (2022)
- [What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions](https://mlsanity.com/paper/2303.16990/) (2023)
- [The 2023 Video Similarity Dataset and Challenge](https://mlsanity.com/paper/2306.09489/) (2023)
- [Learning Segment Similarity and Alignment in Large-Scale Content Based Video Retrieval](https://mlsanity.com/paper/2309.11091/) (2023)
- [Not All Pairs are Equal: Hierarchical Learning for Average-Precision-Oriented Video Retrieval](https://mlsanity.com/paper/2407.15566/) (2024)
- [Let All be Whitened: Multi-teacher Distillation for Efficient Visual Retrieval](https://mlsanity.com/paper/2312.09716/) (2023)
- [The MeVer DeepFake Detection Service: Lessons Learnt from Developing and Deploying in the Wild](https://mlsanity.com/paper/2204.12816/) (2022)
- [VADER: Video Alignment Differencing and Retrieval](https://mlsanity.com/paper/2303.13193/) (2023)
- [Colo-SCRL: Self-Supervised Contrastive Representation Learning for Colonoscopic Video Retrieval](https://mlsanity.com/paper/2303.15671/) (2023)
- [VVS: Video-to-Video Retrieval with Irrelevant Frame Suppression](https://mlsanity.com/paper/2303.08906/) (2023)
- [Self-supervised Video Retrieval Transformer Network](https://mlsanity.com/paper/2104.07993/) (2021)
- [A Fast Partial Video Copy Detection Using KNN and Global Feature Database](https://mlsanity.com/paper/2105.01713/) (2021)
- [3D-CSL: self-supervised 3D context similarity learning for Near-Duplicate Video Retrieval](https://mlsanity.com/paper/2211.05352/) (2022)
- [A Comparative Study of Perceptual Quality Metrics for Audio-driven Talking Head Videos](https://mlsanity.com/paper/2403.06421/) (2024)
- [GTA-HDR: A Large-Scale Synthetic Dataset for HDR Image Reconstruction](https://mlsanity.com/paper/2403.17837/) (2024)
- [Sali4Vid: Saliency-Aware Video Reweighting and Adaptive Caption Retrieval for Dense Video Captioning](https://mlsanity.com/paper/2509.04602/) (2025)
- [VRAG: Region Attention Graphs for Content-Based Video Retrieval](https://mlsanity.com/paper/2205.09068/) (2022)
- [3rd Place Solution to Meta AI Video Similarity Challenge](https://mlsanity.com/paper/2304.11964/) (2023)
- [Differentiable Resolution Compression and Alignment for Efficient Video Classification and Retrieval](https://mlsanity.com/paper/2309.08167/) (2023)
- [Learning Collaborative Knowledge with Multimodal Representation for Polyp Re-Identification](https://mlsanity.com/paper/2408.05914/) (2024)
- [Counteracting temporal attacks in Video Copy Detection](https://mlsanity.com/paper/2501.11171/) (2025)

## Abstract (arXiv)

In this paper we introduce ViSiL, a Video Similarity Learning architecture that considers fine-grained Spatio-Temporal relations between pairs of videos -- such relations are typically lost in previous video retrieval approaches that embed the whole frame or even the whole video into a vector descriptor before the similarity estimation. By contrast, our Convolutional Neural Network (CNN)-based approach is trained to calculate video-to-video similarity from refined frame-to-frame similarity matrices, so as to consider both intra- and inter-frame relations. In the proposed method, pairwise frame similarity is estimated by applying Tensor Dot (TD) followed by Chamfer Similarity (CS) on regional CNN frame features - this avoids feature aggregation before the similarity calculation between frames. Subsequently, the similarity matrix between all video frames is fed to a four-layer CNN, and then summarized using Chamfer Similarity (CS) into a video-to-video similarity score -- this avoids feature aggregation before the similarity calculation between videos and captures the temporal similarity patterns between matching frame sequences. We train the proposed network using a triplet loss scheme and evaluate it on five public benchmark datasets on four different video retrieval problems where we demonstrate large improvements in comparison to the state of the art. The implementation of ViSiL is publicly available.