mlsanityData to 7 Oct 2026

TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval

arXiv 2001.09099 ↗First posted 24 Jan 2020cs.CV cs.CL cs.IRcited by 215

Not on the map. Why?

Abstract (arXiv)

We introduce TV show Retrieval (TVR), a new multimodal retrieval dataset. TVR requires systems to understand both videos and their associated subtitle (dialogue) texts, making it more realistic. The dataset contains 109K queries collected on 21.8K videos from 6 TV shows of diverse genres, where each query is associated with a tight temporal window. The queries are also labeled with query types that indicate whether each of them is more related to video or subtitle or both, allowing for in-depth analysis of the dataset and the methods that built on top of it. Strict qualification and post-annotation verification tests are applied to ensure the quality of the collected data. Further, we present several baselines and a novel Cross-modal Moment Localization (XML ) network for multimodal moment retrieval tasks. The proposed XML model uses a late fusion design with a novel Convolutional Start-End detector (ConvSE), surpassing baselines by a large margin and with better efficiency, providing a strong starting point for future work. We have also collected additional descriptions for each annotated moment in TVR to form a new multimodal captioning dataset with 262K captions, named TV show Caption (TVC). Both datasets are publicly available. TVR: https://tvr.cs.unc.edu, TVC: https://tvr.cs.unc.edu/tvc.html.

Key facts. Cited by 215 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 32; 2022: 32; 2023: 46; 2024: 44; 2025: 40; 2026: 21.

215 papers.

Sort:

By area:

  1. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  2. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  3. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  4. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  5. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  6. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  7. 2023Video Understanding with Large Language Models: A Surveycited by 203
  8. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  9. 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 194
  10. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 189
  11. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  12. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
38 more
  1. 2023Query-Dependent Video Representation for Moment Retrieval and Highlight Detectioncited by 166
  2. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  3. 2022UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detectioncited by 148
  4. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  5. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  6. 2023Hierarchical Video-Moment Retrieval and Step-Captioningcited by 83
  7. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  8. 2023Knowing Where to Focus: Event-aware Transformer for Video Groundingcited by 79
  9. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  10. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  11. 2021QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queriescited by 72
  12. 2021Visual Semantic Role Labeling for Video Understandingcited by 60
  13. 2021Video Corpus Moment Retrieval with Contrastive Learningcited by 60
  14. 2024TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detectioncited by 59
  15. 2023MomentDiff: Generative Video Moment Retrieval from Random to Realcited by 58
  16. 2021Target Adaptive Context Aggregation for Video Scene Graph Generationcited by 51
  17. 2023AutoAD II: The Sequel -- Who, When, and What in Movie Audio Descriptioncited by 48
  18. 2021On Semantic Similarity in Video Retrievalcited by 45
  19. 2022Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Videocited by 43
  20. 2022Multi-Modal Cross-Domain Alignment Network for Video Moment Retrievalcited by 43
  21. 2021VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputscited by 42
  22. 2023You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videoscited by 41
  23. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  24. 2023MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformercited by 40
  25. 2021Cross-Sentence Temporal and Semantic Relations in Video Activity Localisationcited by 39
  26. 2023VidChapters-7M: Video Chapters at Scalecited by 39
  27. 2022Hierarchical Local-Global Transformer for Temporal Sentence Groundingcited by 38
  28. 2024Goldfish: Vision-Language Understanding of Arbitrarily Long Videoscited by 38
  29. 2022Temporal Sentence Grounding in Videos: A Survey and Future Directionscited by 37
  30. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  31. 2024R2-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Groundingcited by 34
  32. 2021CONQUER: Contextual Query-aware Ranking for Video Corpus Moment Retrievalcited by 33
  33. 2023Correlation-Guided Query-Dependency Calibration for Video Temporal Groundingcited by 31
  34. 2022Unsupervised Temporal Video Grounding with Deep Semantic Clusteringcited by 30
  35. 2022Visual Abductive Reasoningcited by 30
  36. 2024Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrievalcited by 30
  37. 2021A Survey on Temporal Sentence Grounding in Videoscited by 29
  38. 2023BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videoscited by 29

50 most cited of 215 All 215 (JSON)