mlsanityData to 7 Oct 2026

All in One: Exploring Unified Video-Language Pre-training

arXiv 2203.07303 ↗First posted 14 Mar 2022cs.CVcited by 171

Not on the map. Why?

Cited by 26 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Mainstream Video-Language Pre-training models citeactbert,clipbert,violet consist of three parts, a video encoder, a text encoder, and a video-text fusion Transformer. They pursue better performance via utilizing heavier unimodal encoders or multimodal fusion Transformers, resulting in increased parameters with lower efficiency in downstream tasks. In this work, we for the first time introduce an end-to-end video-language model, namely textitall-in-one Transformer, that embeds raw video and textual signals into joint representations using a unified backbone architecture. We argue that the unique temporal information of video data turns out to be a key barrier hindering the design of a modality-agnostic Transformer. To overcome the challenge, we introduce a novel and effective token rolling operation to encode temporal representations from video clips in a non-parametric manner. The careful design enables the representation learning of both video-text multimodal inputs and unimodal inputs using a unified backbone model. Our pre-trained all-in-one Transformer is transferred to various downstream video-text tasks after fine-tuning, including text-video retrieval, video-question answering, multiple choice and visual commonsense reasoning. State-of-the-art performances with the minimal model FLOPs on nine datasets demonstrate the superiority of our method compared to the competitive counterparts. The code and pretrained model have been released in https://github.com/showlab/all-in-one.

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). Builds on 55 of its 74 references that are in the map. First posted on arXiv 14 Mar 2022.

Builds on

55 of 74 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
38 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  3. 2017Mixed Precision Trainingcited by 1,135
  4. 2021Video Swin Transformercited by 1,090
  5. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  6. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  7. 2017Dense-Captioning Events in Videoscited by 789
  8. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  9. 2017Localizing Moments in Video with Natural Languagecited by 660
  10. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  11. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  12. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  13. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  14. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  15. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  16. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  17. 2017Temporal Segment Networks for Action Recognition in Videoscited by 279
  18. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  19. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  20. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  21. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  22. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  23. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  24. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  25. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  26. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  27. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  28. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
  29. 2019Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answeringcited by 146
  30. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  31. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  32. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  33. 2020Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learningcited by 74
  34. 2021Object-aware Video-language Pre-training for Retrievalcited by 57
  35. 2020Look Before you Speak: Visually Contextualized Utterancescited by 53
  36. 2021UFO: A UniFied TransfOrmer for Vision-Language Representation Learningcited by 45
  37. 2016Learning Language-Visual Embedding for Movie Understanding with Natural-Languagecited by 41
  38. 2021Co-training Transformer with Videos and Images Improves Action Recognitioncited by 41

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 22; 2023: 68; 2024: 51; 2025: 20; 2026: 10.

171 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  4. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  5. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  6. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  7. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  8. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  9. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  10. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  11. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  12. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
38 more
  1. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 189
  2. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  3. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  4. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  5. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  6. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  7. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  8. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  9. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  10. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  11. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  12. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  13. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  14. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  15. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  16. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  17. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  18. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  19. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  20. 2023Audio-Visual LLM for Video Understandingcited by 76
  21. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  22. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  23. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  24. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  25. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 69
  26. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  27. 2023Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferringcited by 57
  28. 2022MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answeringcited by 55
  29. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  30. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51
  31. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  32. 2023Learning Procedure-aware Video Representation from Instructional Videos and Their Narrationscited by 46
  33. 2022Masked Image Modeling with Denoising Contrastcited by 45
  34. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  35. 2023Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptionscited by 42
  36. 2022Clover: Towards A Unified Video-Language Alignment and Fusion Modelcited by 41
  37. 2024Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sportscited by 41
  38. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41

50 most cited of 171 All 171 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization
  2. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
  3. 2025Exploring the Role of Explicit Temporal Modeling in Multimodal Large Language Models for Video Understanding
  4. 2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
  5. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Tasks
  6. 2026Beyond Language Modeling: An Exploration of Multimodal Pretraining
  7. 2025VideoPerceiver: Enhancing Fine-Grained Temporal Perception in Video Multimodal Large Language Models
  8. 2025Slow-Fast Architecture for Video Multi-Modal Large Language Models