# Learning Segment Similarity and Alignment in Large-Scale Content Based Video Retrieval

arXiv: https://arxiv.org/abs/2309.11091 · HTML: https://mlsanity.com/paper/2309.11091/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 39 references that are in the map.
- First posted on arXiv 20 Sep 2023.

## Builds on (most cited first, 13 of 13)

- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Particular object retrieval with integral max-pooling of CNN activations](https://mlsanity.com/paper/1511.05879/) (2015)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [End-to-end Learning of Deep Visual Representations for Image Retrieval](https://mlsanity.com/paper/1610.07940/) (2016)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [Video Summarization Using Fully Convolutional Sequence Networks](https://mlsanity.com/paper/1805.10538/) (2018)
- [ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning](https://mlsanity.com/paper/1908.07410/) (2019)
- [FIVR: Fine-grained Incident Video Retrieval](https://mlsanity.com/paper/1809.04094/) (2018)
- [Video Re-localization](https://mlsanity.com/paper/1808.01575/) (2018)
- [Circulant temporal encoding for video retrieval and temporal alignment](https://mlsanity.com/paper/1506.02588/) (2015)
- [VSCAN: An Enhanced Video Summarization using Density-based Spatial Clustering](https://mlsanity.com/paper/1405.0174/) (2014)
- [Unsupervised video summarization framework using keyframe extraction and video skimming](https://mlsanity.com/paper/1910.04792/) (2019)

## Built on by (most cited first, 9 of 9)

- [Player-Centric Multimodal Prompt Generation for Large Language Model Based Identity-Aware Basketball Video Captioning](https://mlsanity.com/paper/2507.20163/) (2025)
- [Dynamic Content Moderation in Livestreams: Combining Supervised Classification with MLLM-Boosted Similarity Matching](https://mlsanity.com/paper/2512.03553/) (2025)
- [Knowledge Guided Entity-aware Video Captioning and A Basketball Benchmark](https://mlsanity.com/paper/2401.13888/) (2024)
- [The Effects of Short Video-Sharing Services on Video Copy Detection](https://mlsanity.com/paper/2403.18158/) (2024)
- [Region-aware Image-based Human Action Retrieval with Transformers](https://mlsanity.com/paper/2407.09924/) (2024)
- [Understanding the Performance Plateau in Text-to-Video Retrieval: A Comprehensive Empirical and Linguistic Analysis](https://mlsanity.com/paper/2605.00826/) (2026)
- [MLT-Dedup: Efficient Large-Scale Online Video Deduplication via Multi-Level Representations and Spatial-Temporal Matching](https://mlsanity.com/paper/2606.12215/) (2026)
- [Anomalous Frame Detection by Grouping Frame Similarities between Two Videos Computed by Vision-Language Model to Extract Expert Workers' Unique Actions](https://mlsanity.com/paper/2607.10598/) (2026)
- [Anomalous Frame Detection Using VLM-Based Description Comparison for Extracting Expert-Specific Actions and Contextual Decision-Making Scenes with Intra-Video Self-Similarity](https://mlsanity.com/paper/2607.11957/) (2026)

## Abstract (arXiv)

With the explosive growth of web videos in recent years, large-scale Content-Based Video Retrieval (CBVR) becomes increasingly essential in video filtering, recommendation, and copyright protection. Segment-level CBVR (S-CBVR) locates the start and end time of similar segments in finer granularity, which is beneficial for user browsing efficiency and infringement detection especially in long video scenarios. The challenge of S-CBVR task is how to achieve high temporal alignment accuracy with efficient computation and low storage consumption. In this paper, we propose a Segment Similarity and Alignment Network (SSAN) in dealing with the challenge which is firstly trained end-to-end in S-CBVR. SSAN is based on two newly proposed modules in video retrieval: (1) An efficient Self-supervised Keyframe Extraction (SKE) module to reduce redundant frame features, (2) A robust Similarity Pattern Detection (SPD) module for temporal alignment. In comparison with uniform frame extraction, SKE not only saves feature storage and search time, but also introduces comparable accuracy and limited extra computation time. In terms of temporal alignment, SPD localizes similar segments with higher accuracy and efficiency than existing deep learning methods. Furthermore, we jointly train SSAN with SKE and SPD and achieve an end-to-end improvement. Meanwhile, the two key modules SKE and SPD can also be effectively inserted into other video retrieval pipelines and gain considerable performance improvements. Experimental results on public datasets show that SSAN can obtain higher alignment accuracy while saving storage and online query computational cost compared to existing methods.