mlsanityData to 7 Oct 2026

Localizing Moments in Video with Natural Language

arXiv 1708.01641 ↗First posted 4 Aug 2017cs.CVcited by 660

Not on the map. Why?

Cited by 72 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We consider retrieving a specific temporal segment, or moment, from a video given a natural language text description. Methods designed to retrieve whole video clips with natural language determine what occurs in a video but not when. To address this issue, we propose the Moment Context Network (MCN) which effectively localizes natural language queries in videos by integrating local and global video features over time. A key obstacle to training our MCN model is that current video datasets do not include pairs of localized video segments and referring expressions, or text descriptions which uniquely identify a corresponding moment. Therefore, we collect the Distinct Describable Moments (DiDeMo) dataset which consists of over 10,000 unedited, personal videos in diverse visual settings with pairs of localized video segments and referring expressions. We demonstrate that MCN outperforms several baseline methods and believe that our initial results together with the release of DiDeMo will inspire further research on localizing video moments with natural language.

Key facts. Cited by 660 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Aug 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 82; 2022: 107; 2023: 125; 2024: 120; 2025: 133; 2026: 93.

660 papers.

Sort:

By area:

  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  2. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  3. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  5. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  6. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  7. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  8. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  9. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  10. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  11. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  12. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
38 more
  1. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  2. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  3. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  4. 2022Egocentric Video-Language Pretrainingcited by 231
  5. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  6. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  7. 2023A Survey on Video Diffusion Modelscited by 203
  8. 2023Video Understanding with Large Language Models: A Surveycited by 203
  9. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  10. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  11. 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 194
  12. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  13. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 189
  14. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  15. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  16. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  17. 2023Query-Dependent Video Representation for Moment Retrieval and Highlight Detectioncited by 166
  18. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  19. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  20. 2022UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detectioncited by 148
  21. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  22. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  23. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  24. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  25. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 138
  26. 2021A Survey on Deep Learning Technique for Video Segmentationcited by 132
  27. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  28. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  29. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 124
  30. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  31. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  32. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  33. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  34. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  35. 2021Context-aware Biaffine Localizing Network for Temporal Sentence Groundingcited by 107
  36. 2021MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptionscited by 107
  37. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  38. 2023CelebV-Text: A Large-Scale Facial Text-Video Datasetcited by 103

50 most cited of 660 All 660 (JSON)