mlsanityData to 7 Oct 2026

End-to-end Generative Pretraining for Multimodal Video Captioning

arXiv 2201.08264 ↗First posted 20 Jan 2022cs.CV cs.AI cs.CLcited by 115

Not on the map. Why?

Abstract (arXiv)

Recent video and language pretraining frameworks lack the ability to generate sentences. We present Multimodal Video Generative Pretraining (MV-GPT), a new pretraining framework for learning from unlabelled videos which can be effectively used for generative tasks such as multimodal video captioning. Unlike recent video-language pretraining frameworks, our framework trains both a multimodal video encoder and a sentence decoder jointly. To overcome the lack of captions in unlabelled videos, we leverage the future utterance as an additional text source and propose a bidirectional generation objective -- we generate future utterances given the present mulitmodal context, and also the present utterance given future observations. With this objective, we train an encoder-decoder model end-to-end to generate a caption from raw pixels and transcribed speech directly. Our model achieves state-of-the-art performance for multimodal video captioning on four standard benchmarks, as well as for other video understanding tasks such as VideoQA, video retrieval and action classification.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Builds on 46 of its 65 references that are in the map. First posted on arXiv 20 Jan 2022.

Builds on

46 of 65 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2017The Kinetics Human Action Video Datasetcited by 1,894
  12. 2021ViViT: A Video Vision Transformercited by 1,610
34 more
  1. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  2. 2017Dense-Captioning Events in Videoscited by 789
  3. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  4. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  5. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  6. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  7. 2020Multi-modal Transformer for Video Retrievalcited by 395
  8. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  9. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  10. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  11. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  12. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  13. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  14. 2020Support-set bottlenecks for video-text representation learningcited by 170
  15. 2020Multi-modal Dense Video Captioningcited by 113
  16. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 110
  17. 2020Object Relational Graph with Teacher-Recommended Learning for Video Captioningcited by 106
  18. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  19. 2020MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioningcited by 79
  20. 2020Multimodal Pretraining for Dense Video Captioningcited by 79
  21. 2020A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformercited by 75
  22. 2020Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learningcited by 74
  23. 2018Bidirectional Attentive Fusion with Context Gating for Dense Video Captioningcited by 68
  24. 2018Jointly Localizing and Describing Events for Dense Video Captioningcited by 60
  25. 2020Look Before you Speak: Visually Contextualized Utterancescited by 53
  26. 2019Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Networkcited by 48
  27. 2019Object-aware Aggregation with Bidirectional Temporal Graph for Video Captioningcited by 42
  28. 2019Watch, Listen and Tell: Multi-modal Weakly Supervised Dense Event Captioningcited by 38
  29. 2018Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioningcited by 21
  30. 2020Video Understanding as Machine Translationcited by 20
  31. 2019A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captionscited by 15
  32. 2020Delving Deeper into the Decoder for Video Captioningcited by 13
  33. 2019A Semantics-Assisted Video Captioning Model Trained with Scheduled Samplingcited by 11
  34. 2019Open-Ended Long-Form Video Question Answering via Hierarchical Convolutional Self-Attention Networkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 51; 2024: 33; 2025: 10; 2026: 7.

115 papers.

Sort:

By area:

  1. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  2. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  3. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  4. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  5. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  6. 2022Learning Video Representations from Large Language Modelscited by 204
  7. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  8. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  9. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  10. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  11. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  12. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
38 more
  1. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  2. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  3. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  4. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  5. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  6. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  7. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 69
  8. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generationcited by 63
  9. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  10. 2023SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentariescited by 57
  11. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  12. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  13. 2023AutoAD: Movie Description in Contextcited by 50
  14. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  15. 2023AutoAD II: The Sequel -- Who, When, and What in Movie Audio Descriptioncited by 48
  16. 2024Context-Guided Spatio-Temporal Video Groundingcited by 45
  17. 2023InstructVid2Vid: Controllable Video Editing with Natural Language Instructionscited by 44
  18. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  19. 2023Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptionscited by 42
  20. 2023MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformercited by 40
  21. 2023VidChapters-7M: Video Chapters at Scalecited by 39
  22. 2024Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrievalcited by 36
  23. 2024AutoAD III: The Prequel -- Back to the Pixelscited by 35
  24. 2023HowToCaption: Prompting LLMs to Transform Video Annotations at Scalecited by 34
  25. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 33
  26. 2024Enhancing Video-Language Representations with Structural Spatio-Temporal Alignmentcited by 32
  27. 2023Gloss-Free End-to-End Sign Language Translationcited by 31
  28. 2023GOAL: A Challenging Knowledge-grounded Video Captioning Benchmark for Real-time Soccer Commentary Generationcited by 28
  29. 2023A Review of Deep Learning for Video Captioningcited by 28
  30. 2023Text with Knowledge Graph Augmented Transformer for Video Captioningcited by 27
  31. 2023Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Modelscited by 27
  32. 2023ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Modelscited by 26
  33. 2023MaMMUT: A Simple Architecture for Joint Learning for MultiModal Taskscited by 24
  34. 2023Accurate and Fast Compressed Video Captioningcited by 21
  35. 2024AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Descriptioncited by 21
  36. 2023Language-Guided Audio-Visual Source Separation via Trimodal Consistencycited by 20
  37. 2023VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blendingcited by 17
  38. 2024HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipationcited by 17

50 most cited of 115 All 115 (JSON)