# 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities

arXiv: https://arxiv.org/abs/2406.09406 · HTML: https://mlsanity.com/paper/2406.09406/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 89 of its 105 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 30 of 89)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)

## Built on by (most cited first, 30 of 52)

- [OmniGen: Unified Image Generation](https://mlsanity.com/paper/2409.11340/) (2024)
- [SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models](https://mlsanity.com/paper/2407.15841/) (2024)
- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [DINO-Foresight: Looking into the Future with DINO](https://mlsanity.com/paper/2412.11673/) (2024)
- [DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks](https://mlsanity.com/paper/2502.17157/) (2025)
- [PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions](https://mlsanity.com/paper/2409.15278/) (2024)
- [The Power of Context: How Multimodality Improves Image Super-Resolution](https://mlsanity.com/paper/2503.14503/) (2025)
- [Towards Enhanced Image Generation Via Multi-modal Chain of Thought in Unified Generative Models](https://mlsanity.com/paper/2503.01298/) (2025)
- [How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks](https://mlsanity.com/paper/2507.01955/) (2025)
- [Video Generation Models are General-Purpose Vision Learners](https://mlsanity.com/paper/2607.09024/) (2026)
- [Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions](https://mlsanity.com/paper/2412.08737/) (2024)
- [MMGen: Unified Multi-modal Image Generation and Understanding in One Go](https://mlsanity.com/paper/2503.20644/) (2025)
- [JointDiT: Enhancing RGB-Depth Joint Modeling with Diffusion Transformers](https://mlsanity.com/paper/2505.00482/) (2025)
- [UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark](https://mlsanity.com/paper/2603.05075/) (2026)
- [EgoM2P: Egocentric Multimodal Multitask Pretraining](https://mlsanity.com/paper/2506.07886/) (2025)
- [Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers](https://mlsanity.com/paper/2501.08303/) (2025)
- [On Fairness of Unified Multimodal Large Language Model for Image Generation](https://mlsanity.com/paper/2502.03429/) (2025)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [THFM: A Unified Video Foundation Model for 4D Human Perception and Beyond](https://mlsanity.com/paper/2603.25892/) (2026)
- [DenoMAE: A Multimodal Autoencoder for Denoising Modulation Signals](https://mlsanity.com/paper/2501.11538/) (2025)
- [BiFold: Bimanual Cloth Folding with Language Guidance](https://mlsanity.com/paper/2501.16458/) (2025)
- [SPEAR: A Simulator for Photorealistic Embodied AI Research](https://mlsanity.com/paper/2607.06701/) (2026)
- [Towards a Science Exocortex](https://mlsanity.com/paper/2406.17809/) (2024)
- [The Moon's Many Faces: A Single Unified Transformer for Multimodal Lunar Reconstruction](https://mlsanity.com/paper/2505.05644/) (2025)
- [FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens](https://mlsanity.com/paper/2506.03096/) (2025)
- [Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractor](https://mlsanity.com/paper/2507.07106/) (2025)
- [Probabilistic Precipitation Nowcasting with Rectified Flow Transformers](https://mlsanity.com/paper/2605.31204/) (2026)
- [JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation](https://mlsanity.com/paper/2607.01768/) (2026)
- [ARCON: Advancing Auto-Regressive Continuation for Driving Videos](https://mlsanity.com/paper/2412.03758/) (2024)
- [ModalFormer: Multimodal Transformer for Low-Light Image Enhancement](https://mlsanity.com/paper/2507.20388/) (2025)

## Abstract (arXiv)

Current multimodal and multitask foundation models like 4M or UnifiedIO show promising results, but in practice their out-of-the-box abilities to accept diverse inputs and perform diverse tasks are limited by the (usually rather small) number of modalities and tasks they are trained on. In this paper, we expand upon the capabilities of them by training a single model on tens of highly diverse modalities and by performing co-training on large-scale multimodal datasets and text corpora. This includes training on several semantic and geometric modalities, feature maps from recent state of the art models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM and 4DHumans, and a range of new modalities that allow for novel ways to interact with the model and steer the generation, for example image metadata or color palettes. A crucial step in this process is performing discrete tokenization on various modalities, whether they are image-like, neural network feature maps, vectors, structured data like instance segmentation or human poses, or data that can be represented as text. Through this, we expand on the out-of-the-box capabilities of multimodal models and specifically show the possibility of training one model to solve at least 3x more tasks/modalities than existing ones and doing so without a loss in performance. This enables more fine-grained and controllable multimodal generation capabilities and allows us to study the distillation of models trained on diverse data and objectives into a unified model. We successfully scale the training to a three billion parameter model using tens of modalities and different datasets. The resulting models and training code are open sourced at 4m.epfl.ch.