# Learning video retrieval models with relevance-aware online mining

arXiv: https://arxiv.org/abs/2203.08688 · HTML: https://mlsanity.com/paper/2203.08688/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [Use What You Have: Video Retrieval Using Representations From Collaborative Experts](https://mlsanity.com/paper/1907.13487/) (2019)
- [Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning](https://mlsanity.com/paper/2003.00392/) (2020)
- [Beyond triplet loss: a deep quadruplet network for person re-identification](https://mlsanity.com/paper/1704.01719/) (2017)
- [VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples](https://mlsanity.com/paper/2103.05905/) (2021)
- [Learning a Text-Video Embedding from Incomplete and Heterogeneous Data](https://mlsanity.com/paper/1804.02516/) (2018)
- [T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval](https://mlsanity.com/paper/2104.10054/) (2021)
- [Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings](https://mlsanity.com/paper/1908.03477/) (2019)
- [Smart Mining for Deep Metric Learning](https://mlsanity.com/paper/1704.01285/) (2017)
- [Hard negative examples are hard, but useful](https://mlsanity.com/paper/2007.12749/) (2020)
- [TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrieval](https://mlsanity.com/paper/2104.08271/) (2021)
- [MDMMT: Multidomain Multimodal Transformer for Video Retrieval](https://mlsanity.com/paper/2103.10699/) (2021)
- [Improved Embeddings with Easy Positive Triplet Mining](https://mlsanity.com/paper/1904.04370/) (2019)
- [On Semantic Similarity in Video Retrieval](https://mlsanity.com/paper/2103.10095/) (2021)

## Built on by (most cited first, 6 of 6)

- [A Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval](https://mlsanity.com/paper/2208.02080/) (2022)
- [UniUD-FBK-UB-UniBZ Submission to the EPIC-Kitchens-100 Multi-Instance Retrieval Challenge 2022](https://mlsanity.com/paper/2206.10903/) (2022)
- [An Overview of Challenges in Egocentric Text-Video Retrieval](https://mlsanity.com/paper/2306.04345/) (2023)
- [Leveraging Generative Language Models for Weakly Supervised Sentence Component Analysis in Video-Language Joint Learning](https://mlsanity.com/paper/2312.06699/) (2023)
- [UniUD Submission to the EPIC-Kitchens-100 Multi-Instance Retrieval Challenge 2023](https://mlsanity.com/paper/2306.15445/) (2023)
- [Towards Debiasing Frame Length Bias in Text-Video Retrieval via Causal Intervention](https://mlsanity.com/paper/2309.09311/) (2023)

## Abstract (arXiv)

Due to the amount of videos and related captions uploaded every hour, deep learning-based solutions for cross-modal video retrieval are attracting more and more attention. A typical approach consists in learning a joint text-video embedding space, where the similarity of a video and its associated caption is maximized, whereas a lower similarity is enforced with all the other captions, called negatives. This approach assumes that only the video and caption pairs in the dataset are valid, but different captions - positives - may also describe its visual contents, hence some of them may be wrongly penalized. To address this shortcoming, we propose the Relevance-Aware Negatives and Positives mining (RANP) which, based on the semantics of the negatives, improves their selection while also increasing the similarity of other valid positives. We explore the influence of these techniques on two video-text datasets: EPIC-Kitchens-100 and MSR-VTT. By using the proposed techniques, we achieve considerable improvements in terms of nDCG and mAP, leading to state-of-the-art results, e.g. +5.3% nDCG and +3.0% mAP on EPIC-Kitchens-100. We share code and pretrained models at urlhttps://github.com/aranciokov/ranp.