mlsanityData to 7 Oct 2026

MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts

arXiv 2407.21770 ↗First posted 31 Jul 2024cs.AI cs.LGcited by 61

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

We introduce MoMa, a novel modality-aware mixture-of-experts (MoE) architecture designed for pre-training mixed-modal, early-fusion language models. MoMa processes images and text in arbitrary sequences by dividing expert modules into modality-specific groups. These groups exclusively process designated tokens while employing learned routing within each group to maintain semantically informed adaptivity. Our empirical results reveal substantial pre-training efficiency gains through this modality-specific parameter allocation. Under a 1-trillion-token training budget, the MoMa 1.4B model, featuring 4 text experts and 4 image experts, achieves impressive FLOPs savings: 3.7x overall, with 2.6x for text and 5.2x for image processing compared to a compute-equivalent dense baseline, measured by pre-training loss. This outperforms the standard expert-choice MoE with 8 mixed-modal experts, which achieves 3x overall FLOPs savings (3x for text, 2.8x for image). Combining MoMa with mixture-of-depths (MoD) further improves pre-training FLOPs savings to 4.2x overall (text: 3.4x, image: 5.3x), although this combination hurts performance in causal inference due to increased sensitivity to router accuracy. These results demonstrate MoMa's potential to significantly advance the efficiency of mixed-modal, early-fusion language model pre-training, paving the way for more resource-efficient and capable multimodal AI systems.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 33 of its 42 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

33 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts6 Jun

Other references · 32

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
20 more
  1. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  2. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  3. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  5. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  6. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  7. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  8. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  9. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  10. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  11. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  12. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  13. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  14. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  15. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  16. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  17. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  18. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  19. 2021Gumbel-Attention for Multi-modal Machine Translationcited by 20
  20. 2020How Does Selective Mechanism Improve Self-Attention Networks?cited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 27; 2026: 19.

61 papers.

Sort:

By area:

  1. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  2. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  3. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  4. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  5. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  6. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136
  7. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  8. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  9. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  10. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  11. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 38
  12. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
38 more
  1. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 32
  2. 2024EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routingcited by 29
  3. 2025I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Expertscited by 29
  4. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 27
  5. 2024Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registerscited by 19
  6. 2025OmniCaptioner: One Captioner to Rule Them Allcited by 19
  7. 2025Watermarking Autoregressive Image Generationcited by 15
  8. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scalecited by 15
  9. 2025Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Modelscited by 13
  10. 2025R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expertscited by 12
  11. 2025Optimus-3: Dual-Router Aligned Mixture-of-Experts Agent with Dual-Granularity Reasoning-Aware Policy Optimizationcited by 12
  12. 2025Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoningcited by 12
  13. 2024Mixture of Experts in Image Classification: What's the Sweet Spot?cited by 11
  14. 2025Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsitycited by 7
  15. 2024Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Modelscited by 6
  16. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  17. 2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Modelcited by 6
  18. 2025COLLAGE: Adaptive Fusion-based Retrieval for Augmented Policy Learningcited by 5
  19. 2025Flexible Multitask Learning with Factorized Diffusion Policycited by 5
  20. 2025UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depthscited by 4
  21. 2025MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Modelscited by 4
  22. 2025CoSMoEs: Compact Sparse Mixture of Expertscited by 3
  23. 2025MAPEX: Modality-Aware Pruning of Experts for Remote Sensing Foundation Modelscited by 3
  24. 2025MoSE: Skill-by-Skill Mixture-of-Experts Learning for Embodied Autonomous Machinescited by 3
  25. 2025End-to-end Listen, Look, Speak and Actcited by 3
  26. 2026Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancingcited by 3
  27. 2026Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Trainingcited by 3
  28. 2026SenseNova-U1.5: Towards Native Unified Visual Intelligencecited by 3
  29. 2024LaB-RAG: Label Boosted Retrieval Augmented Generation for Radiology Report Generationcited by 2
  30. 2025AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Modelscited by 2
  31. 2026MoST: Mixing Speech and Text with Modality-Aware Mixture of Expertscited by 2
  32. 2025Multimodal Long Video Modeling Based on Temporal Dynamic Contextcited by 1
  33. 2025DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoEcited by 1
  34. 2025Multimodal Language Models with Modality-Specific Experts for Financial Forecasting from Interleaved Sequences of Text and Time Seriescited by 1
  35. 2025SigLino: Efficient Multi-Teacher Distillation for Agglomerative Vision Foundation Modelscited by 1
  36. 2026Improving MoE Compute Efficiency by Composing Weight and Data Sparsitycited by 1
  37. 2026Multi-Modal Time Series Prediction via Mixture of Modulated Expertscited by 1
  38. 2026ReaLB: Real-Time Load Balancing for Multimodal MoE Inferencecited by 1

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities
  2. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models
  3. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts
  4. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts
  5. 2026MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
  6. 2025Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilities
  7. 2026EMO: Pretraining Mixture of Experts for Emergent Modularity
  8. 2026Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models