mlsanityData to 7 Oct 2026

CM3: A Causal Masked Multimodal Model of the Internet

arXiv 2201.07520 ↗First posted 19 Jan 2022cs.CLcited by 142

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce CM3, a family of causally masked generative models trained over a large corpus of structured multi-modal documents that can contain both text and image tokens. Our new causally masked approach generates tokens left to right while also masking out a small number of long token spans that are generated at the end of the string, instead of their original positions. The casual masking object provides a type of hybrid of the more common causal and masked language models, by enabling full generative modeling while also providing bidirectional context when generating the masked spans. We train causally masked language-image models on large-scale web and Wikipedia articles, where each document contains all of the text, hypertext markup, hyperlinks, and image tokens (from a VQVAE-GAN), provided in the order they appear in the original HTML source (before masking). The resulting CM3 models can generate rich structured, multi-modal outputs while conditioning on arbitrary masked document contexts, and thereby implicitly learn a wide range of text, image, and cross modal tasks. They can be prompted to recover, in a zero-shot fashion, the functionality of models such as DALL-E, GENRE, and HTLM. We set the new state-of-the-art in zero-shot summarization, entity linking, and entity disambiguation while maintaining competitive performance in the fine-tuning setting. We can generate images unconditionally, conditioned on text (like DALL-E) and do captioning all in a zero-shot setting with a single model.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). Builds on 50 of its 68 references that are in the map. First posted on arXiv 19 Jan 2022.

Builds on

50 of 68 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  11. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2015Teaching Machines to Read and Comprehendcited by 1,039
  9. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  10. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  11. 2020Jukebox: A Generative Model for Musiccited by 535
  12. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  13. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  14. 2020Autoregressive Entity Retrievalcited by 333
  15. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  16. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  17. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  18. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  19. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  20. 2019Assessing Social and Intersectional Biases in Contextualized Word Representationscited by 134
  21. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  22. 2018Abstractive Summarization of Reddit Posts with Multi-level Memory Networkscited by 102
  23. 2018End-to-End Neural Entity Linkingcited by 85
  24. 2017Deep Joint Entity Disambiguation with Local Neural Attentioncited by 79
  25. 2021Improving Text-to-Image Synthesis Using Contrastive Learningcited by 62
  26. 2020Measuring Social Biases in Grounded Vision and Language Embeddingscited by 61
  27. 2020REL: An Entity Linker Standing on the Shoulders of Giantscited by 61
  28. 2021HTLM: Hyper-Text Pre-Training and Prompting of Language Modelscited by 58
  29. 2018Improving Entity Linking by Modeling Latent Relations between Mentionscited by 53
  30. 2020Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentationcited by 53
  31. 2020Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linkingcited by 49
  32. 2019Joint Entity Linking with Deep Reinforcement Learningcited by 27
  33. 2019Joint Learning of Named Entity Recognition and Entity Linkingcited by 26
  34. 2014Analysis of Named Entity Recognition and Linking for Tweetscited by 20
  35. 2019Learning Dynamic Context Augmentation for Global Entity Linkingcited by 19
  36. 2018Collective Entity Disambiguation with Structured Gradient Tree Boostingcited by 16
  37. 2019Boosting Entity Linking Performance by Leveraging Unlabeled Documentscited by 12
  38. 2019Entity-aware ELMo: Learning Contextual Entity Representation for Entity Disambiguationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 24; 2023: 59; 2024: 39; 2025: 15; 2026: 5.

142 papers.

Sort:

By area:

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  5. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  6. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  7. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  8. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  9. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  10. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  11. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  12. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
38 more
  1. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  2. 2022CodeT: Code Generation with Generated Testscited by 369
  3. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  4. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  5. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  6. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  7. 2023Generating Images with Multimodal Language Modelscited by 259
  8. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  9. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  10. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  11. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  12. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  13. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  14. 2023Language-Driven Representation Learning for Roboticscited by 163
  15. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  16. 2022A Review on Language Models as Knowledge Basescited by 156
  17. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  18. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  19. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  20. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  21. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  22. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  23. 20234M: Massively Multimodal Masked Modelingcited by 118
  24. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  25. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  26. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  27. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  28. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  29. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 98
  30. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  31. 2023GIVT: Generative Infinite-Vocabulary Transformerscited by 80
  32. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  33. 2023IconShop: Text-Guided Vector Icon Synthesis with Autoregressive Transformerscited by 73
  34. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  35. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 61
  36. 2023Identifiability Results for Multimodal Contrastive Learningcited by 58
  37. 2025Edit Flows: Flow Matching with Edit Operationscited by 53
  38. 2023An Empirical Study of Multimodal Model Mergingcited by 52

50 most cited of 142 All 142 (JSON)