mlsanityData to 7 Oct 2026

Connecting Vision and Language with Video Localized Narratives

arXiv 2302.11217 ↗First posted 22 Feb 2023cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

We propose Video Localized Narratives, a new form of multimodal video annotations connecting vision and language. In the original Localized Narratives, annotators speak and move their mouse simultaneously on an image, thus grounding each word with a mouse trace segment. However, this is challenging on a video. Our new protocol empowers annotators to tell the story of a video with Localized Narratives, capturing even complex events involving multiple actors interacting with each other and with several passive objects. We annotated 20k videos of the OVIS, UVO, and Oops datasets, totalling 1.7M words. Based on this data, we also construct new benchmarks for the video narrative grounding and video question answering tasks, and provide reference results from strong baseline models. Our annotations are available at https://google.github.io/video-localized-narratives/.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 54 of its 67 references that are in the map. First posted on arXiv 22 Feb 2023.

Builds on

54 of 67 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2017The Kinetics Human Action Video Datasetcited by 1,894
38 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  5. 2021Video Swin Transformercited by 1,090
  6. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  7. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  8. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  9. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  10. 2017Dense-Captioning Events in Videoscited by 789
  11. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  13. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  14. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  15. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  16. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  17. 2020Rescaling Egocentric Visioncited by 529
  18. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  19. 2019Connecting Vision and Language with Localized Narrativescited by 225
  20. 2022Language as Queries for Referring Video Object Segmentationcited by 180
  21. 2018Video Object Segmentation with Language Referring Expressionscited by 161
  22. 2021Occluded Video Instance Segmentation: A Benchmarkcited by 154
  23. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
  24. 2021Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentationcited by 133
  25. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  26. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  27. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  28. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  29. 2019Oops! Predicting Unintentional Action in Videocited by 82
  30. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  31. 2020Multimodal Pretraining for Dense Video Captioningcited by 79
  32. 2018Grounded Video Descriptioncited by 78
  33. 2022All You May Need for VQA are Image Captionscited by 65
  34. 2016Leveraging Video Descriptions to Learn Video Question Answeringcited by 55
  35. 2021Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptionscited by 53
  36. 2020DramaQA: Character-Centered Video Story Understanding with Hierarchical QAcited by 45
  37. 2018Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interactioncited by 40
  38. 2017Video Question Answering via Attribute-Augmented Attention Network Learningcited by 39

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 8; 2025: 10; 2026: 1.

24 papers.

Sort:

By area:

  1. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  2. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  3. 2023Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videoscited by 84
  4. 2024VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Modelscited by 40
  5. 2023Voila-A: Aligning Vision-Language Models with User's Gaze Attentioncited by 28
  6. 2023StoryBench: A Multifaceted Benchmark for Continuous Story Visualizationcited by 27
  7. 2024Distilling Vision-Language Models on Millions of Videoscited by 20
  8. 2024ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentationcited by 16
  9. 2023Dense Video Object Captioning from Disjoint Supervisioncited by 10
  10. 2024Toward Interactive Regional Understanding in Vision-Large Language Modelscited by 9
  11. 2025Make Your Training Flexible: Towards Deployment-Efficient Video Modelscited by 6
  12. 2025InternVideo-Next: Towards General Video Foundation Models without Video-Text Supervisioncited by 6
12 more
  1. 2024Point-VOS: Pointing Up Video Object Segmentationcited by 5
  2. 2025What Are You Doing? A Closer Look at Controllable Human Video Generationcited by 3
  3. 2025Extending Visual Dynamics for Video-to-Music Generationcited by 3
  4. 2025VoCap: Video Object Captioning and Segmentation from Any Promptcited by 3
  5. 2025MedicalNarratives: Connecting Medical Vision and Language with Localized Narrativescited by 2
  6. 2025A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospectscited by 2
  7. 2025FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentationcited by 2
  8. 2024Beyond Coarse-Grained Matching in Video-Text Retrievalcited by 1
  9. 2026TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understandingcited by 1
  10. 2024Think Before You Act: A Two-Stage Framework for Mitigating Gender Bias Towards Vision-Language Taskscited by 0
  11. 2025Resolving Ambiguity in Gaze-Facilitated Visual Assistant Interaction Paradigmcited by 0
  12. 2025VideoChain: A Transformer-Based Framework for Multi-hop Video Question Generationcited by 0