mlsanityData to 7 Oct 2026

MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound

arXiv 2201.02639 ↗First posted 7 Jan 2022cs.CV cs.CL cs.LGcited by 184

Not on the map. Why?

Cited by 26 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As humans, we navigate a multimodal world, building a holistic understanding from all our senses. We introduce MERLOT Reserve, a model that represents videos jointly over time -- through a new training objective that learns from audio, subtitles, and video frames. Given a video, we replace snippets of text and audio with a MASK token; the model learns by choosing the correct masked-out snippet. Our objective learns faster than alternatives, and performs well at scale: we pretrain on 20 million YouTube videos. Empirical results show that MERLOT Reserve learns strong multimodal representations. When finetuned, it sets state-of-the-art on Visual Commonsense Reasoning (VCR), TVQA, and Kinetics-600; outperforming prior work by 5%, 7%, and 1.5% respectively. Ablations show that these tasks benefit from audio pretraining -- even VCR, a QA task centered around images (without sound). Moreover, our objective enables out-of-the-box prediction, revealing strong multimodal commonsense understanding. In a fully zero-shot setting, our model obtains competitive results on four video tasks, even outperforming supervised approaches on the recently proposed Situated Reasoning (STAR) benchmark. We analyze why audio enables better vision-language representations, suggesting significant opportunities for future research. We conclude by discussing ethical and societal implications of multimodal pretraining.

Key facts. Cited by 184 papers in the map (data through 7 Oct 2026). Builds on 87 of its 136 references that are in the map. First posted on arXiv 7 Jan 2022.

Builds on

87 of 136 references.

References · 87

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2020Language Models are Few-Shot Learnerscited by 32,199
  7. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2020Extracting Training Data from Large Language Modelscited by 2,144
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2018Datasheets for Datasetscited by 1,371
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  13. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  14. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  15. 2021Scaling Vision Transformerscited by 993
  16. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  17. 2017VGGFace2: A dataset for recognising faces across pose and agecited by 911
  18. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  19. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  20. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  21. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  22. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  23. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  24. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  25. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  26. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  27. 2021AST: Audio Spectrogram Transformercited by 634
  28. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  29. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  30. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  31. 2019Defending Against Neural Fake Newscited by 553
  32. 2020Rescaling Egocentric Visioncited by 529
  33. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  34. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  35. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  36. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  37. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  38. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 29; 2023: 75; 2024: 39; 2025: 31; 2026: 10.

184 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  4. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  6. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  7. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  8. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  9. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  10. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  11. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  12. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
38 more
  1. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  2. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  3. 2022Matryoshka Representation Learningcited by 268
  4. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  5. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  6. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  7. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  8. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  9. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  10. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  11. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  12. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  13. 2022FlexiViT: One Model for All Patch Sizescited by 151
  14. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  15. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  16. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  17. 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoningcited by 122
  18. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  19. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  20. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  21. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  22. 2023Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videoscited by 84
  23. 2024Streaming Dense Video Captioningcited by 82
  24. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  25. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  26. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  27. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  28. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generationcited by 63
  29. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  30. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 58
  31. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 57
  32. 2022MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answeringcited by 55
  33. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understandingcited by 55
  34. 2022Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learningcited by 49
  35. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 49
  36. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  37. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  38. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46

50 most cited of 184 All 184 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Unifying Specialized Visual Encoders for Video Language Models
  2. 2026MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos
  3. 2026How Far Are Video Models from True Multimodal Reasoning?
  4. 2026OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video
  5. 2026LEMON: How Well Do MLLMs Perform Temporal Multimodal Understanding on Instructional Videos?
  6. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs
  7. 2026MCSC-Bench: Multimodal Context-to-Script Creation for Realistic Video Production ↗
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding