mlsanityData to 7 Oct 2026

On Semantic Similarity in Video Retrieval

arXiv 2103.10095 ↗First posted 18 Mar 2021cs.CVcited by 45

Not on the map. Why?

Abstract (arXiv)

Current video retrieval efforts all found their evaluation on an instance-based assumption, that only a single caption is relevant to a query video and vice versa. We demonstrate that this assumption results in performance comparisons often not indicative of models' retrieval capabilities. We propose a move to semantic similarity video retrieval, where (i) multiple videos/captions can be deemed equally relevant, and their relative ranking does not affect a method's reported performance and (ii) retrieved videos/captions are ranked by their similarity to a query. We propose several proxies to estimate semantic similarities in large-scale retrieval datasets, without additional annotations. Our analysis is performed on three commonly used video retrieval datasets (MSR-VTT, YouCook2 and EPIC-KITCHENS).

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 47 of its 78 references that are in the map. First posted on arXiv 18 Mar 2021.

Builds on

47 of 78 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  7. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  8. 2017Dense-Captioning Events in Videoscited by 789
  9. 2017Localizing Moments in Video with Natural Languagecited by 660
  10. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  11. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  12. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
35 more
  1. 2020Multi-modal Transformer for Video Retrievalcited by 395
  2. 2020A Metric Learning Reality Checkcited by 317
  3. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  4. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  5. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
  6. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  7. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  8. 2015A Dataset for Movie Descriptioncited by 195
  9. 2019Learning with Average Precision: Training Image Retrieval with a Listwise Losscited by 184
  10. 2020Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoningcited by 180
  11. 2020Support-set bottlenecks for video-text representation learningcited by 170
  12. 2015Sequence to Sequence -- Video to Textcited by 157
  13. 2019REPAIR: Removing Representation Bias by Dataset Resamplingcited by 157
  14. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 150
  15. 2019Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognitioncited by 117
  16. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 110
  17. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  18. 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
  19. 2019Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddingscited by 101
  20. 2020Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrievalcited by 98
  21. 2020Condensed Movies: Story Based Retrieval with Contextual Embeddingscited by 95
  22. 2015Describing Videos by Exploiting Temporal Structurecited by 88
  23. 2018Diagnosing Error in Temporal Action Detectorscited by 71
  24. 2020Uncovering Hidden Challenges in Query-Based Video Moment Retrievalcited by 58
  25. 2016Leveraging Video Descriptions to Learn Video Question Answeringcited by 55
  26. 2017Predicting Visual Features from Text for Image and Video Caption Retrievalcited by 53
  27. 2016Learning Language-Visual Embedding for Movie Understanding with Natural-Languagecited by 41
  28. 2018Hierarchy-based Image Embeddings for Semantic Image Retrievalcited by 41
  29. 2019Deep Metric Learning Beyond Binary Supervisioncited by 36
  30. 2019Action Modifiers: Learning from Adverbs in Instructional Videoscited by 27
  31. 2016Learning Joint Representations of Videos and Sentences with Web Image Searchcited by 17
  32. 2017Trespassing the Boundaries: Labeling Temporal Bounds for Object Interactions in Egocentric Videocited by 14
  33. 2020The End-of-End-to-End: A Video Understanding Pentathlon Challenge (2020)cited by 6
  34. 2020QuerYD: A video dataset with high-quality text and audio narrationscited by 6
  35. 2020Dual Encoding for Video Retrieval by Textcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 19; 2023: 8; 2024: 6; 2025: 7; 2026: 2.

45 papers.

Sort:

By area:

  1. 2022Egocentric Video-Language Pretrainingcited by 231
  2. 2022Temporal Alignment Networks for Long-term Videocited by 84
  3. 2023Knowing Where to Focus: Event-aware Transformer for Video Groundingcited by 79
  4. 2022CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detectioncited by 67
  5. 2022Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representationscited by 57
  6. 2022ECLIPSE: Efficient Long-range Video Retrieval using Sight and Soundcited by 39
  7. 2025ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shortscited by 36
  8. 2022Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrievalcited by 34
  9. 2022Probabilistic Representations for Video Contrastive Learningcited by 33
  10. 2021HANet: Hierarchical Alignment Networks for Video-Text Retrievalcited by 31
  11. 2023Towards Video Anomaly Retrieval from Video Anomaly Detection: New Benchmarks and Modelcited by 25
  12. 2022Text-Adaptive Multiple Visual Prototype Matching for Video-Text Retrievalcited by 20
33 more
  1. 2025Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrievalcited by 20
  2. 2022Integrating Language Guidance into Vision-based Deep Metric Learningcited by 19
  3. 2022Learning to Refactor Action and Co-occurrence Features for Temporal Action Localizationcited by 18
  4. 2023Collaborative Group: Composed Image Retrieval via Consensus Learning from Noisy Annotationscited by 14
  5. 2022A Feature-space Multimodal Data Augmentation Technique for Text-video Retrievalcited by 13
  6. 2024Reversed in Time: A Novel Temporal-Emphasized Benchmark for Cross-Modal Video-Text Retrievalcited by 12
  7. 2022Multi-Query Video Retrievalcited by 11
  8. 2021Domain Adaptation in Multi-View Embedding for Cross-Modal Video Retrievalcited by 6
  9. 2022Learning video retrieval models with relevance-aware online miningcited by 6
  10. 2022Relevance-based Margin for Contrastively-trained Video Retrieval Modelscited by 6
  11. 2025NeighborRetr: Balancing Hub Centrality in Cross-Modal Retrievalcited by 6
  12. 2025What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learningcited by 6
  13. 2022Exploiting Semantic Role Contextualized Video Features for Multi-Instance Text-Video Retrieval EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022cited by 5
  14. 2022Contextual Explainable Video Representation: Human Perception-based Understandingcited by 5
  15. 2023Mask to reconstruct: Cooperative Semantics Completion for Video-text Retrievalcited by 5
  16. 2025Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learningcited by 4
  17. 2021SwAMP: Swapped Assignment of Multi-Modal Pairs for Cross-Modal Retrievalcited by 3
  18. 2025EVA02-AT: Egocentric Video-Language Understanding with Spatial-Temporal Rotary Positional Embeddings and Symmetric Optimizationcited by 3
  19. 2022Towards Privacy-Preserving, Real-Time and Lossless Feature Matchingcited by 2
  20. 2022Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarkscited by 1
  21. 2022Image-Text Retrieval with Binary and Continuous Label Supervisioncited by 1
  22. 2023An Overview of Challenges in Egocentric Text-Video Retrievalcited by 1
  23. 2023A Video is Worth 10,000 Words: Training and Benchmarking with Diverse Captions for Better Long Video Retrievalcited by 1
  24. 2024CarcassFormer: An End-to-end Transformer-based Framework for Simultaneous Localization, Segmentation and Classification of Poultry Carcass Defectcited by 1
  25. 2024Beyond Coarse-Grained Matching in Video-Text Retrievalcited by 1
  26. 2023UniUD Submission to the EPIC-Kitchens-100 Multi-Instance Retrieval Challenge 2023cited by 0
  27. 2023Towards Debiasing Frame Length Bias in Text-Video Retrieval via Causal Interventioncited by 0
  28. 2024Image Similarity using An Ensemble of Context-Sensitive Modelscited by 0
  29. 2024CBVS: A Large-Scale Chinese Image-Text Benchmark for Real-World Short Video Search Scenarioscited by 0
  30. 2024Towards Holistic Language-video Representation: the language model-enhanced MSR-Video to Text Datasetcited by 0
  31. 2025EgoVIS@CVPR: What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learningcited by 0
  32. 2026EagleNet: Energy-Aware Fine-Grained Relationship Learning Network for Text-Video Retrievalcited by 0
  33. 2026Understanding the Performance Plateau in Text-to-Video Retrieval: A Comprehensive Empirical and Linguistic Analysiscited by 0