mlsanityData to 13 Oct 2026

MERLOT: Multimodal Neural Script Knowledge Models

arXiv 2106.02636 ↗First posted 4 Jun 2021cs.CV cs.CL cs.LGcited by 311

Not on the map. Why?

Cited by 37 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As humans, we understand events in the visual world contextually, performing multimodal reasoning across time to make inferences about the past, present, and future. We introduce MERLOT, a model that learns multimodal script knowledge by watching millions of YouTube videos with transcribed speech -- in an entirely label-free, self-supervised manner. By pretraining with a mix of both frame-level (spatial) and video-level (temporal) objectives, our model not only learns to match images to temporally corresponding words, but also to contextualize what is happening globally over time. As a result, MERLOT exhibits strong out-of-the-box representations of temporal commonsense, and achieves state-of-the-art performance on 12 different video QA datasets when finetuned. It also transfers well to the world of static images, allowing models to reason about the dynamic context behind visual scenes. On Visual Commonsense Reasoning, MERLOT answers questions correctly with 80.6% accuracy, outperforming state-of-the-art models of similar size by over 3%, even those that make heavy use of auxiliary supervised data (like object bounding boxes). Ablation analyses demonstrate the complementary importance of: 1) training on videos versus static images; 2) scaling the magnitude and diversity of the pretraining video corpus; and 3) using diverse objectives that encourage full-stack multimodal reasoning, from the recognition to cognition level.

Key facts. Cited by 311 papers in the map (data through 13 Oct 2026). Builds on 80 of its 132 references that are in the map. First posted on arXiv 4 Jun 2021.

Builds on

80 of 132 references.

References · 80

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,625
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,786
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,910
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,893
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,649
38 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,220
  2. 2020Extracting Training Data from Large Language Modelscited by 2,181
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,551
  4. 2018Datasheets for Datasetscited by 1,377
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,294
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,248
  8. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,227
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 999
  10. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  11. 2019Attention is not Explanationcited by 922
  12. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 833
  13. 2017Dense-Captioning Events in Videoscited by 790
  14. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  15. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 639
  16. 2019Attention is not not Explanationcited by 633
  17. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 560
  18. 2019Defending Against Neural Fake Newscited by 559
  19. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  20. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  21. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 505
  22. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  23. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  24. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  25. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 428
  26. 2019Is Attention Interpretable?cited by 388
  27. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  28. 2016Visual Relationship Detection with Language Priorscited by 333
  29. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  30. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  31. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  32. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  33. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 259
  34. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  35. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 227
  36. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  37. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  38. 2016Movie Descriptioncited by 180

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 11; 2022: 71; 2023: 113; 2024: 67; 2025: 34; 2026: 15.

311 papers.

Sort:

By area:

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,321
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,052
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 854
  6. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 764
  7. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 739
  8. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 567
  9. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  10. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 492
  11. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 446
  12. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 404
38 more
  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 390
  3. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  4. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  5. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  6. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  7. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 216
  8. 2023A Survey on Video Diffusion Modelscited by 206
  9. 2022Learning Video Representations from Large Language Modelscited by 204
  10. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  11. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  12. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  13. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  14. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 186
  15. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  16. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 184
  17. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  18. 2022Revisiting the "Video" in Video-Language Understandingcited by 168
  19. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  20. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 162
  21. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  22. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  23. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 154
  24. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  25. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  26. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  27. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 134
  28. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 134
  29. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 132
  30. 2023Large Language Models for Robotics: A Surveycited by 132
  31. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  32. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  33. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 120
  34. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  35. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  36. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 113
  37. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  38. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103

50 most cited of 311 All 311 (JSON)