mlsanityData to 14 Oct 2026

VideoOFA: Two-Stage Pre-Training for Video-to-Text Generation

arXiv 2305.03204 ↗First posted 4 May 2023cs.CV cs.CLcited by 3

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

We propose a new two-stage pre-training framework for video-to-text generation tasks such as video captioning and video question answering: A generative encoder-decoder model is first jointly pre-trained on massive image-text data to learn fundamental vision-language concepts, and then adapted to video data in an intermediate video-text pre-training stage to learn video-specific skills such as spatio-temporal reasoning. As a result, our VideoOFA model achieves new state-of-the-art performance on four Video Captioning benchmarks, beating prior art by an average of 9.7 points in CIDEr score. It also outperforms existing models on two open-ended Video Question Answering datasets, showcasing its generalization capability as a universal video-to-text model.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 51 of its 62 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

51 of 62 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2021ViViT: A Video Vision Transformercited by 1,620
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
38 more
  1. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  2. 2021Video Swin Transformercited by 1,091
  3. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  4. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  5. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  6. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  7. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  8. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  9. 2016Self-critical Sequence Training for Image Captioningcited by 547
  10. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  11. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  12. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  13. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  14. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  15. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  16. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  17. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  18. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  19. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  20. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  21. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  22. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 162
  23. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  24. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 116
  25. 2020Spatio-Temporal Graph for Video Captioning with Knowledge Distillationcited by 109
  26. 2020Object Relational Graph with Teacher-Recommended Learning for Video Captioningcited by 107
  27. 2014Translating Videos to Natural Language Using Deep Recurrent Neural Networkscited by 95
  28. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  29. 2020A Short Note on the Kinetics-700-2020 Human Action Datasetcited by 92
  30. 2015Describing Videos by Exploiting Temporal Structurecited by 88
  31. 2021CLIP4Caption: CLIP for Video Captioncited by 88
  32. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  33. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  34. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  35. 2019Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioningcited by 60
  36. 2020Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videoscited by 56
  37. 2016Leveraging Video Descriptions to Learn Video Question Answeringcited by 55
  38. 2021Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptionscited by 53

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2024An Introduction to Vision-Language Modelingcited by 178
  2. 2023Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalitiescited by 15
  3. 2024Whats in a Video: Factorized Autoregressive Decoding for Online Dense Video Captioningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Distilling Vision-Language Models on Millions of Videos
  2. 2026Advancing Video-Text Pretraining with Multi-View Captions
  3. 2025TextVidBench: A Benchmark for Long Video Scene Text Understanding
  4. 2023GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generation
  5. 2025TV2TV: A Unified Framework for Interleaved Language and Video Generation
  6. 2025Foundational Question Generation for Video Question Answering via an Embedding-Integrated Approach
  7. 2025FIQ: Fundamental Question Generation with the Integration of Question Embeddings for Video Question Answering
  8. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answering