mlsanityData to 14 Oct 2026

MultiMAE: Multi-modal Multi-task Masked Autoencoders

arXiv 2204.01678 ↗First posted 4 Apr 2022cs.CV cs.LGcited by 291

Not on the map. Why?

Abstract (arXiv)

We propose a pre-training strategy called Multi-modal Multi-task Masked Autoencoders (MultiMAE). It differs from standard Masked Autoencoding in two key aspects: I) it can optionally accept additional modalities of information in the input besides the RGB image (hence "multi-modal"), and II) its training objective accordingly includes predicting multiple outputs besides the RGB image (hence "multi-task"). We make use of masking (across image patches and input modalities) to make training MultiMAE tractable as well as to ensure cross-modality predictive coding is indeed learned by the network. We show this pre-training strategy leads to a flexible, simple, and efficient framework with improved transfer results to downstream tasks. In particular, the same exact pre-trained network can be flexibly used when additional information besides RGB images is available or when no information other than RGB is available - in all configurations yielding competitive to or significantly better results than the baselines. To avoid needing training datasets with multiple modalities and tasks, we train MultiMAE entirely using pseudo labeling, which makes the framework widely applicable to any RGB dataset. The experiments are performed on multiple transfer tasks (image classification, semantic segmentation, depth estimation) and datasets (ImageNet, ADE20K, Taskonomy, Hypersim, NYUv2). The results show an intriguingly impressive capability by the model in cross-modal/task predictive coding and transfer.

Key facts. Cited by 291 papers in the map (data through 14 Oct 2026). Builds on 78 of its 102 references that are in the map. First posted on arXiv 4 Apr 2022.

Builds on

78 of 102 references.

References · 78

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2016Layer Normalizationcited by 5,669
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  12. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2022A ConvNet for the 2020scited by 3,579
  4. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  7. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  10. 2021Vision Transformers for Dense Predictioncited by 1,803
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  13. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  14. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  15. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  16. 2019Natural Adversarial Examplescited by 1,408
  17. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  18. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  19. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  20. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  21. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  22. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  23. 2016Deep Networks with Stochastic Depthcited by 979
  24. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  25. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  26. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  27. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  28. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  29. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  30. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  31. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  32. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 604
  33. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 596
  34. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  35. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  36. 2017Look, Listen and Learncited by 458
  37. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 437
  38. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 32; 2023: 77; 2024: 67; 2025: 66; 2026: 49.

291 papers.

Sort:

By area:

  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  2. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  3. 2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 408
  4. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  5. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  6. 2025MapAnything: Universal Feed-Forward Metric 3D Reconstructioncited by 314
  7. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  8. 2022Masked World Models for Visual Controlcited by 170
  9. 2022CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flowcited by 166
  10. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 160
  11. 2022Contrastive Audio-Visual Masked Autoencodercited by 146
  12. 20234M: Massively Multimodal Masked Modelingcited by 120
38 more
  1. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  2. 2023Delivering Arbitrary-Modal Semantic Segmentationcited by 117
  3. 2024MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learningcited by 77
  4. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  5. 2024Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentationcited by 71
  6. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 71
  7. 2024A Touch, Vision, and Language Dataset for Multimodal Alignmentcited by 69
  8. 2022Masked Vision and Language Modeling for Multi-modal Representation Learningcited by 67
  9. 2023Identifiability Results for Multimodal Contrastive Learningcited by 61
  10. 2023DFormer: Rethinking RGBD Representation Learning for Semantic Segmentationcited by 60
  11. 2022MAViL: Masked Audio-Video Learnerscited by 59
  12. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  13. 2023Hard Patches Mining for Masked Image Modelingcited by 57
  14. 2023PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigmcited by 56
  15. 2022Towards Good Practices for Missing Modality Robust Action Recognitioncited by 54
  16. 2023PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detectioncited by 53
  17. 2024Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biologycited by 52
  18. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  19. 2022Audiovisual Masked Autoencoderscited by 49
  20. 2024What to align in multimodal contrastive learning?cited by 47
  21. 2024Bridging Remote Sensors with Multisensor Geospatial Foundation Modelscited by 46
  22. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 46
  23. 2023TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Expertscited by 45
  24. 2024Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learningcited by 45
  25. 2024Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Taskscited by 45
  26. 2023Traj-MAE: Masked Autoencoders for Trajectory Predictioncited by 42
  27. 2023An Unbiased Look at Datasets for Visuo-Motor Pre-Trainingcited by 42
  28. 2022Masked Event Modeling: Self-Supervised Pretraining for Event Camerascited by 41
  29. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  30. 2025Matrix3D: Large Photogrammetry Model All-in-Onecited by 40
  31. 2024Cubify Anything: Scaling Indoor 3D Object Detectioncited by 39
  32. 2023Self-Distilled Masked Auto-Encoders are Efficient Video Anomaly Detectorscited by 37
  33. 2025TAPNext: Tracking Any Point (TAP) as Next Token Predictioncited by 37
  34. 2022Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detectioncited by 33
  35. 2022XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learningcited by 31
  36. 2025DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentationcited by 31
  37. 2024Cross-View Completion Models are Zero-shot Correspondence Estimatorscited by 30
  38. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 29

50 most cited of 291 All 291 (JSON)