mlsanityData to 14 Oct 2026

VideoBERT: A Joint Model for Video and Language Representation Learning

arXiv 1904.01766 ↗First posted 3 Apr 2019cs.CV cs.AIcited by 703

Not on the map. Why?

Cited by 55 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube. Whereas most existing approaches learn low-level representations, we propose a joint visual-linguistic model to learn high-level features without any explicit supervision. In particular, inspired by its recent success in language modeling, we build upon the BERT model to learn bidirectional joint distributions over sequences of visual and linguistic tokens, derived from vector quantization of video data and off-the-shelf speech recognition outputs, respectively. We use VideoBERT in numerous tasks, including action classification and video captioning. We show that it can be applied directly to open-vocabulary classification, and confirm that large amounts of training data and cross-modal information are critical to performance. Furthermore, we outperform the state-of-the-art on video captioning, and quantitative results verify that the model learns high-level semantic features.

Key facts. Cited by 703 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 177; 2022: 179; 2023: 147; 2024: 96; 2025: 70; 2026: 34.

703 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  4. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  5. 2021Transformers in Vision: A Surveycited by 1,013
  6. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  7. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  9. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  10. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  11. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  12. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
38 more
  1. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  2. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 485
  3. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 485
  4. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  5. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  6. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  7. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  8. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  9. 2021Dynamic Neural Networks: A Surveycited by 408
  10. 2021A Survey of Transformerscited by 401
  11. 2022Multimodal Learning with Transformers: A Surveycited by 400
  12. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 377
  13. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  14. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  15. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  16. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  17. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  18. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
  19. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  20. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  21. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  22. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  23. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 280
  24. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  25. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  26. 2021Understanding Robustness of Transformers for Image Classificationcited by 249
  27. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  28. 2022Egocentric Video-Language Pretrainingcited by 232
  29. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  30. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  31. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  32. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  33. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  34. 2023Video Understanding with Large Language Models: A Surveycited by 206
  35. 2022Learning Video Representations from Large Language Modelscited by 205
  36. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 203
  37. 2021Anticipative Video Transformercited by 200
  38. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196

50 most cited of 703 All 703 (JSON)