mlsanityData to 7 Oct 2026

Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning

arXiv 2302.14115 ↗First posted 27 Feb 2023cs.CV cs.AI cs.CLcited by 263

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale. The Vid2Seq architecture augments a language model with special time tokens, allowing it to seamlessly predict event boundaries and textual descriptions in the same output sequence. Such a unified model requires large-scale training data, which is not available in current annotated datasets. We show that it is possible to leverage unlabeled narrated videos for dense video captioning, by reformulating sentence boundaries of transcribed speech as pseudo event boundaries, and using the transcribed speech sentences as pseudo event captions. The resulting Vid2Seq model pretrained on the YT-Temporal-1B dataset improves the state of the art on a variety of dense video captioning benchmarks including YouCook2, ViTT and ActivityNet Captions. Vid2Seq also generalizes well to the tasks of video paragraph captioning and video clip captioning, and to few-shot settings. Our code is publicly available at https://antoyang.github.io/vid2seq.html.

Key facts. Cited by 263 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 115 of its 136 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

115 of 136 references.

References · 115

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
38 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  7. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  8. 2021Grounded Language-Image Pre-trainingcited by 1,147
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  11. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  12. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  13. 2017Dense-Captioning Events in Videoscited by 789
  14. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  15. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  16. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  17. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  18. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  19. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  20. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  21. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  22. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  23. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  24. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  25. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  26. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  27. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  28. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  29. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  30. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  31. 2019BMN: Boundary-Matching Network for Temporal Action Proposal Generationcited by 327
  32. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  33. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  34. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  35. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 308
  36. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  37. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  38. 2018BSN: Boundary Sensitive Network for Temporal Action Proposal Generationcited by 271

50 most cited of 115 All 115 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 42; 2024: 93; 2025: 86; 2026: 42.

263 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  3. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  4. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  5. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  6. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  7. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  8. 2023Video Understanding with Large Language Models: A Surveycited by 203
  9. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 189
  10. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  11. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 138
  12. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
38 more
  1. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Modelscited by 124
  2. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  3. 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoningcited by 122
  4. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  5. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  6. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 89
  7. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  8. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  9. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 87
  10. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  11. 2024Streaming Dense Video Captioningcited by 82
  12. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  13. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 72
  14. 2023UnLoc: A Unified Framework for Video Localization Taskscited by 66
  15. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  16. 2023SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentariescited by 57
  17. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  18. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  19. 2025ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understandingcited by 50
  20. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 49
  21. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  22. 2023AutoAD II: The Sequel -- Who, When, and What in Movie Audio Descriptioncited by 48
  23. 2024Context-Guided Spatio-Temporal Video Groundingcited by 45
  24. 2024V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuningcited by 45
  25. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 44
  26. 2023Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptionscited by 42
  27. 2024WorldGPT: Empowering LLM as Multimodal World Modelcited by 42
  28. 2023VidChapters-7M: Video Chapters at Scalecited by 39
  29. 2024When Large Language Models Meet Vector Databases: A Surveycited by 39
  30. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
  31. 2024A Survey on Generative AI and LLM for Video Generation, Understanding, and Streamingcited by 37
  32. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  33. 2024MatchTime: Towards Automatic Soccer Game Commentary Generationcited by 37
  34. 2024Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrievalcited by 36
  35. 2024AutoAD III: The Prequel -- Back to the Pixelscited by 35
  36. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  37. 2023HowToCaption: Prompting LLMs to Transform Video Annotations at Scalecited by 34
  38. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 34

50 most cited of 263 All 263 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning
  2. 2026CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning
  3. 2026Advancing Video-Text Pretraining with Multi-View Captions
  4. 2026DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Models
  5. 2025VidChain: Chain-of-Tasks with Metric-based Direct Preference Optimization for Dense Video Captioning
  6. 2025IF-VidCap: Can Video Caption Models Follow Instructions?
  7. 2025Toward Scalable Video Narration: A Training-free Approach Using Multimodal Large Language Models
  8. 2024Cap2Sum: Learning to Summarize Videos by Generating Captions