mlsanityData to 7 Oct 2026

Scaling Laws for Native Multimodal Models

arXiv 2504.07951 ↗First posted 10 Apr 2025cs.CVcited by 47

Not on the map. Why?

Abstract (arXiv)

Building general-purpose models that can effectively perceive the world through multimodal signals has been a long-standing goal. Current approaches involve integrating separately pre-trained components, such as connecting vision encoders to LLMs and continuing multimodal training. While such approaches exhibit remarkable sample efficiency, it remains an open question whether such late-fusion architectures are inherently superior. In this work, we revisit the architectural design of native multimodal models (NMMs)-those trained from the ground up on all modalities-and conduct an extensive scaling laws study, spanning 457 trained models with different architectures and training mixtures. Our investigation reveals no inherent advantage to late-fusion architectures over early-fusion ones, which do not rely on image encoders or tokenizers. On the contrary, early-fusion exhibits stronger performance at lower parameter counts, is more efficient to train, and is easier to deploy. Motivated by the strong performance of the early-fusion architectures, we show that incorporating Mixture of Experts (MoEs) allows models to learn modality-specific weights, significantly benefiting performance.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 80 of its 91 references that are in the map. First posted on arXiv 10 Apr 2025.

Builds on

80 of 91 references.

References · 80

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2024GPT-4o System Cardcited by 4,467
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
38 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2024Mixtral of Expertscited by 1,785
  11. 2020GLU Variants Improve Transformercited by 1,724
  12. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  13. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  14. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  15. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  16. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  17. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  18. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  19. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  20. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  21. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  22. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  23. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  24. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  25. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  26. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  27. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  28. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  29. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  30. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  31. 2024What matters when building vision-language models?cited by 321
  32. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  33. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  34. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  35. 2022Masked Autoencoders that Listencited by 271
  36. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  37. 2023Data Filtering Networkscited by 247
  38. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 26; 2026: 21.

47 papers.

Sort:

By area:

  1. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  2. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  3. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  4. 2025Scaling Laws for Optimal Data Mixturescited by 49
  5. 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generationcited by 45
  6. 2025Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Modelcited by 38
  7. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 32
  8. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-trainingcited by 23
  9. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 17
  10. 2025Watermarking Autoregressive Image Generationcited by 15
  11. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scalecited by 15
  12. 2025Learning to Steer: Input-dependent Steering for Multimodal LLMscited by 13
35 more
  1. 2025NanoVLA: Routing Decoupled Vision-Language Understanding for Nano-sized Generalist Robotic Policiescited by 12
  2. 2026MKA: Memory-Keyed Attention for Efficient Long-Context Reasoningcited by 6
  3. 2026LongCat-Flash-Prover: Advancing Native Formal Reasoning via Agentic Tool-Integrated Reinforcement Learningcited by 6
  4. 2025Compute-Optimal Scaling for Value-Based Deep RLcited by 5
  5. 2025NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraintscited by 5
  6. 2025Fusion or Confusion? Multimodal Complexity Is Not All You Needcited by 4
  7. 2025RL makes MLLMs see better than SFTcited by 3
  8. 2026Visual Enhanced Depth Scaling for Multimodal Latent Reasoningcited by 3
  9. 2025SeqPE: Transformer with Sequential Position Encodingcited by 2
  10. 2025Evaluating the Robustness of Chinchilla Compute-Optimal Scalingcited by 2
  11. 2025Learning to Generate Rigid Body Interactions with Video Diffusion Modelscited by 2
  12. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 2
  13. 2025Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Modelscited by 2
  14. 2026AnySlot: Goal-Conditioned Vision-Language-Action Policies for Zero-Shot Slot-Level Placementcited by 2
  15. 2026ZAYA1-VL-8B Technical Reportcited by 2
  16. 2026Scaling Native Multimodal Pre-Training From Scratchcited by 2
  17. 2025SigLino: Efficient Multi-Teacher Distillation for Agglomerative Vision Foundation Modelscited by 1
  18. 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipescited by 1
  19. 2025Beyond the Linear Separability Ceiling: Aligning Representations in VLMscited by 0
  20. 2025PiERN: Token-Level Routing for Integrating High-Precision Computation and Reasoningcited by 0
  21. 2025InstructPLM-mu: 1-Hour Fine-Tuning of ESM2 Beats ESM3 in Protein Mutation Predictionscited by 0
  22. 2025Relative-Based Scaling Law for Neural Language Modelscited by 0
  23. 2025A Multimodal Transformer Approach for UAV Detection and Aerial Object Recognition Using Radar, Audio, and Video Datacited by 0
  24. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  25. 2026MARS: Harmonizing Multimodal Convergence via Adaptive Rank Searchcited by 0
  26. 2026Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fitscited by 0
  27. 2026Lifting Unlabeled Internet-level Data for 3D Scene Understandingcited by 0
  28. 2026On the Invariance and Generality of Neural Scaling Lawscited by 0
  29. 2026TPS-Drive: Task-Guided Representation Purification for VLM-based Autonomous Drivingcited by 0
  30. 2026What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMscited by 0
  31. 2026How Far Can 5,500 Hours of Driving Take You? A Scaling Law Analysis of Video Diffusion Modelscited by 0
  32. 2026Do VLMs Share Safety Neurons Across Modalities?cited by 0
  33. 2026Studying Image Tokenizers as Visual Languages in Unified Multimodal Modelscited by 0
  34. 2026How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretrainingcited by 0
  35. 2026Paired Multimodal Scaling Lawscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Toward Native Multimodal Modeling: A Roadmap
  2. 2025Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective
  3. 2025Understanding and Harnessing Sparsity in Unified Multimodal Models
  4. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
  5. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models
  6. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
  7. 2025Dynamic Modality Scheduling for Multimodal Large Models via Confidence, Uncertainty, and Semantic Consistency
  8. 2026A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models