mlsanityData to 7 Oct 2026

Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models

arXiv 2411.04996 ↗First posted 7 Nov 2024cs.CLcited by 151

Not on the map. Why?

Cited by 36 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The development of large language models (LLMs) has expanded to multi-modal systems capable of processing text, images, and speech within a unified framework. Training these models demands significantly larger datasets and computational resources compared to text-only LLMs. To address the scaling challenges, we introduce Mixture-of-Transformers (MoT), a sparse multi-modal transformer architecture that significantly reduces pretraining computational costs. MoT decouples non-embedding parameters of the model by modality -- including feed-forward networks, attention matrices, and layer normalization -- enabling modality-specific processing with global self-attention over the full input sequence. We evaluate MoT across multiple settings and model scales. In the Chameleon 7B setting (autoregressive text-and-image generation), MoT matches the dense baseline's performance using only 55.8% of the FLOPs. When extended to include speech, MoT reaches speech performance comparable to the dense baseline with only 37.2% of the FLOPs. In the Transfusion setting, where text and image are trained with different objectives, a 7B MoT model matches the image modality performance of the dense baseline with one third of the FLOPs, and a 760M MoT model outperforms a 1.4B dense baseline across key image generation metrics. System profiling further highlights MoT's practical benefits, achieving dense baseline image quality in 47.2% of the wall-clock time and text quality in 75.6% of the wall-clock time (measured on AWS p4de.24xlarge instances with NVIDIA A100 GPUs).

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). Builds on 54 of its 67 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

54 of 67 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Classifier-Free Diffusion Guidancecited by 5,717
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
38 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  10. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  11. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  12. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  13. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  14. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  15. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  16. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  17. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  18. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  19. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  20. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  21. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  22. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  23. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  24. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  25. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  26. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  27. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  28. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  29. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  30. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  31. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 113
  32. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  33. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 98
  34. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  35. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 61
  36. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 55
  37. 2024MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesiscited by 44
  38. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-trainingcited by 37

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 45; 2026: 105.

151 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  2. 2026Causal World Modeling for Robot Controlcited by 289
  3. 2025Motus: A Unified Latent Action World Modelcited by 231
  4. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  5. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Bettercited by 142
  6. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  7. 2025GR-3 Technical Reportcited by 110
  8. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  9. 2025F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actionscited by 85
  10. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  11. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 52
  12. 2025DepthVLA: Enhancing Vision-Language-Action Models with Depth-Aware Spatial Reasoningcited by 44
38 more
  1. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 43
  2. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  3. 2026Xiaomi-Robotics-0: An Open-Sourced Vision-Language-Action Model with Real-Time Executioncited by 42
  4. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  5. 2025Interleaving Reasoning for Better Text-to-Image Generationcited by 40
  6. 2025G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoningcited by 39
  7. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 38
  8. 2025MotionGPT3: Human Motion as a Second Modalitycited by 36
  9. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 32
  10. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 30
  11. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 27
  12. 2026Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectoriescited by 24
  13. 2026UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervisioncited by 23
  14. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learningcited by 23
  15. 2026InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editingcited by 23
  16. 2026Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stackcited by 20
  17. 2025SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Modelscited by 19
  18. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  19. 2026HY-Embodied-0.5: Embodied Foundation Models for Real-World Agentscited by 18
  20. 2025OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flowscited by 17
  21. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 17
  22. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Drivingcited by 17
  23. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 17
  24. 2025X-Streamer: Unified Human World Modeling with Audiovisual Interactioncited by 16
  25. 2025Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesiscited by 15
  26. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizercited by 15
  27. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scalecited by 15
  28. 2026MaskWAM: Unifying Mask Prompting and Prediction for World-Action Modelscited by 15
  29. 2025Towards Enhanced Image Generation Via Multi-modal Chain of Thought in Unified Generative Modelscited by 14
  30. 2025Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generationcited by 14
  31. 2026Next Forcing: Causal World Modeling with Multi-Chunk Predictioncited by 14
  32. 2026Faster-WAM: Efficient Inference-Time Future Conditioning for Robust World Action Modelscited by 14
  33. 2025Video-As-Prompt: Unified Semantic Control for Video Generationcited by 13
  34. 2025ManualVLA: A Unified VLA Model for Chain-of-Thought Manual Generation and Robotic Manipulationcited by 13
  35. 2026HALO: A Unified Vision-Language-Action Model for Embodied Multimodal Chain-of-Thought Reasoningcited by 13
  36. 2026MemoryWAM: Efficient World Action Modeling with Persistent Memorycited by 12
  37. 2026Wall-OSS-0.5 Technical Reportcited by 11
  38. 2025TBAC-UniImage: Unified Understanding and Generation by Ladder-Side Diffusion Tuningcited by 10

50 most cited of 151 All 151 (JSON)