mlsanityData to 7 Oct 2026

Multimodal Pretraining for Dense Video Captioning

arXiv 2011.11760 ↗First posted 10 Nov 2020cs.CV cs.CL cs.LGcited by 79

Not on the map. Why?

Abstract (arXiv)

Learning specific hands-on skills such as cooking, car maintenance, and home repairs increasingly happens via instructional videos. The user experience with such videos is known to be improved by meta-information such as time-stamped annotations for the main steps involved. Generating such annotations automatically is challenging, and we describe here two relevant contributions. First, we construct and release a new dense video captioning dataset, Video Timeline Tags (ViTT), featuring a variety of instructional videos together with time-stamped annotations. Second, we explore several multimodal sequence-to-sequence pretraining strategies that leverage large unsupervised datasets of videos and caption-like texts. We pretrain and subsequently finetune dense video captioning models using both YouCook2 and ViTT. We show that such models generalize well and are robust over a wide variety of instructional videos.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 9; 2023: 16; 2024: 24; 2025: 22; 2026: 4.

79 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  3. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  4. 2023Video Understanding with Large Language Models: A Surveycited by 203
  5. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  6. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 115
  7. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  8. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  9. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
  10. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  11. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  12. 2024Streaming Dense Video Captioningcited by 82
38 more
  1. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  2. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  3. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  4. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  5. 2024Online Video Understanding: OVBench and VideoChat-Onlinecited by 58
  6. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  7. 2025LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Callingcited by 58
  8. 2023SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentariescited by 57
  9. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 57
  10. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  11. 2023AutoAD: Movie Description in Contextcited by 50
  12. 2023AutoAD II: The Sequel -- Who, When, and What in Movie Audio Descriptioncited by 48
  13. 2024V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuningcited by 45
  14. 2023VidChapters-7M: Video Chapters at Scalecited by 39
  15. 2024Mind the Time: Temporally-Controlled Multi-Event Video Generationcited by 39
  16. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  17. 2024AutoAD III: The Prequel -- Back to the Pixelscited by 35
  18. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  19. 2023A Review of Deep Learning for Video Captioningcited by 28
  20. 2022End-to-end Dense Video Captioning as Sequence Generationcited by 27
  21. 2025Streaming Video Instruction Tuningcited by 25
  22. 2023Connecting Vision and Language with Video Localized Narrativescited by 24
  23. 2021Masking Modalities for Cross-modal Video Retrievalcited by 23
  24. 2023Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocolscited by 17
  25. 2021SVIP: Sequence VerIfication for Procedures in Videoscited by 16
  26. 2022Video Question Answering with Iterative Video-Text Co-Tokenizationcited by 16
  27. 2024Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videoscited by 15
  28. 2021Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Visioncited by 14
  29. 2023Skip-Plan: Procedure Planning in Instructional Videos via Condensed Action Space Learningcited by 14
  30. 2022FindIt: Generalized Localization with Natural Language Queriescited by 13
  31. 2024JRDB-Social: A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social Groupscited by 13
  32. 2024StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identificationcited by 13
  33. 2025TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Actioncited by 13
  34. 2022Answer-Me: Multi-Task Open-Vocabulary Visual Question Answeringcited by 12
  35. 2025Enrich and Detect: Video Temporal Grounding with Multimodal LLMscited by 12
  36. 2024A Survey of Video Datasets for Grounded Event Understandingcited by 9
  37. 2025Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMscited by 9
  38. 2024Implicit Location-Caption Alignment via Complementary Masking for Weakly-Supervised Dense Video Captioningcited by 8

50 most cited of 79 All 79 (JSON)