mlsanityData to 14 Oct 2026

4M: Massively Multimodal Masked Modeling

arXiv 2312.06647 ↗First posted 11 Dec 2023cs.CV cs.AI cs.LGcited by 120

Not on the map. Why?

Abstract (arXiv)

Current machine learning models for vision are often highly specialized and limited to a single modality and task. In contrast, recent large language models exhibit a wide range of capabilities, hinting at a possibility for similarly versatile models in computer vision. In this paper, we take a step in this direction and propose a multimodal training scheme called 4M. It consists of training a single unified Transformer encoder-decoder using a masked modeling objective across a wide range of input/output modalities - including text, images, geometric, and semantic modalities, as well as neural network feature maps. 4M achieves scalability by unifying the representation space of all modalities through mapping them into discrete tokens and performing multimodal masked modeling on a small randomized subset of tokens. 4M leads to models that exhibit several key capabilities: (1) they can perform a diverse set of vision tasks out of the box, (2) they excel when fine-tuned for unseen downstream tasks or new input modalities, and (3) they can function as a generative model that can be conditioned on arbitrary modalities, enabling a wide variety of expressive multimodal editing capabilities with remarkable flexibility. Through experimental analyses, we demonstrate the potential of 4M for training versatile and scalable foundation models for vision tasks, setting the stage for further exploration in multimodal learning for vision and other domains.

Key facts. Cited by 120 papers in the map (data through 14 Oct 2026). Builds on 116 of its 133 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

116 of 133 references.

References · 116

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  8. 2023GPT-4 Technical Reportcited by 18,396
  9. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  10. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  11. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  12. 2020Denoising Diffusion Implicit Modelscited by 9,781
38 more
  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2017Mask R-CNNcited by 7,277
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2022Classifier-Free Diffusion Guidancecited by 5,847
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2017Neural Discrete Representation Learningcited by 4,712
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  13. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  14. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  15. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  16. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  17. 2022A ConvNet for the 2020scited by 3,579
  18. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  19. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  20. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  21. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  22. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  23. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  24. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  25. 2020GLU Variants Improve Transformercited by 1,804
  26. 2021Vision Transformers for Dense Predictioncited by 1,803
  27. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  28. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  29. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  30. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  31. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  32. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  33. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  34. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  35. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  36. 2023PaLM 2 Technical Reportcited by 1,162
  37. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  38. 2022MaskGIT: Masked Generative Image Transformercited by 1,099

50 most cited of 116 All 116 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 39; 2025: 46; 2026: 35.

120 papers.

Sort:

By area:

  1. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  2. 2024Poseidon: Efficient Foundation Models for PDEscited by 200
  3. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 125
  4. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 94
  5. 2024MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learningcited by 77
  6. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 69
  7. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  8. 2024What Makes Multimodal In-Context Learning Work?cited by 56
  9. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  10. 2025Scaling Laws for Native Multimodal Modelscited by 50
  11. 2024What to align in multimodal contrastive learning?cited by 47
  12. 2024JetFormer: An Autoregressive Generative Model of Raw Images and Textcited by 46
38 more
  1. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 45
  2. 2025Matrix3D: Large Photogrammetry Model All-in-Onecited by 40
  3. 2025DICEPTION: A Generalist Diffusion Model for Visual Perceptual Taskscited by 39
  4. 2025Halton Scheduler For Masked Generative Image Transformercited by 32
  5. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  6. 2024Unraveling Molecular Structure: A Multimodal Spectroscopic Dataset for Chemistrycited by 31
  7. 2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
  8. 2025AToken: A Unified Tokenizer for Visioncited by 30
  9. 2024Masked Image Modeling: A Surveycited by 29
  10. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 23
  11. 2024The Evolution of Multimodal Model Architecturescited by 21
  12. 2024Multi-Transmotion: Pre-trained Model for Human Motion Predictioncited by 21
  13. 2025Neural Encoding and Decoding at Scalecited by 20
  14. 2024A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpaintingcited by 18
  15. 2024Wearable Accelerometer Foundation Models for Health via Knowledge Distillationcited by 16
  16. 2025RealGeneral: Unifying Visual Generation via Temporal In-Context Learning with Video Modelscited by 16
  17. 2025The Power of Context: How Multimodality Improves Image Super-Resolutioncited by 15
  18. 2025Are Large Brainwave Foundation Models Capable Yet? Insights from Fine-tuningcited by 15
  19. 2025TerraMesh: A Planetary Mosaic of Multimodal Earth Observation Datacited by 14
  20. 2025How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Taskscited by 14
  21. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  22. 2026Video Generation Models are General-Purpose Vision Learnerscited by 14
  23. 2024UniTable: Towards a Unified Framework for Table Recognition via Self-Supervised Pretrainingcited by 13
  24. 2024AdaNAT: Exploring Adaptive Policy for Token-Based Image Generationcited by 13
  25. 2024Promoting cross-modal representations to improve multimodal foundation models for physiological signalscited by 13
  26. 2024XQ-GAN: An Open-source Image Tokenization Framework for Autoregressive Generationcited by 12
  27. 2025MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videoscited by 12
  28. 2025Image Tokenizer Needs Post-Trainingcited by 12
  29. 2024Global atmospheric data assimilation with multi-modal masked autoencoderscited by 11
  30. 2025MMGen: Unified Multi-modal Image Generation and Understanding in One Gocited by 11
  31. 2025Robust Latent Matters: Boosting Image Generation with Sampling Error Synthesiscited by 10
  32. 2025EgoM2P: Egocentric Multimodal Multitask Pretrainingcited by 9
  33. 2025SMARTIES: Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Imagescited by 9
  34. 2025Visual Instruction Pretraining for Domain-Specific Foundation Modelscited by 9
  35. 2024PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representationcited by 8
  36. 2024ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesiscited by 8
  37. 2025Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformerscited by 8
  38. 2026Context Unrolling in Omni Modelscited by 8

50 most cited of 120 All 120 (JSON)