mlsanityData to 7 Oct 2026

Scaling Laws for Generative Mixed-Modal Language Models

arXiv 2301.03728 ↗First posted 10 Jan 2023cs.CL cs.AI cs.LGcited by 131

Not on the map. Why?

Abstract (arXiv)

Generative language models define distributions over sequences of tokens that can represent essentially any combination of data modalities (e.g., any permutation of image tokens from VQ-VAEs, speech tokens from HuBERT, BPE tokens for language or code, and so on). To better understand the scaling properties of such mixed-modal models, we conducted over 250 experiments using seven different modalities and model sizes ranging from 8 million to 30 billion, trained on 5-100 billion tokens. We report new mixed-modal scaling laws that unify the contributions of individual modalities and the interactions between them. Specifically, we explicitly model the optimal synergy and competition due to data and model size as an additive term to previous uni-modal scaling laws. We also find four empirical phenomena observed during the training, such as emergent coordinate-ascent style training that naturally alternates between modalities, guidelines for selecting critical hyper-parameters, and connections between mixed-modal competition and training stability. Finally, we test our scaling law by training a 30B speech-text model, which significantly outperforms the corresponding unimodal models. Overall, our research provides valuable insights into the design and training of mixed-modal generative models, an important new class of unified models that have unique distributional properties.

Key facts. Cited by 131 papers in the map (data through 7 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 10 Jan 2023.

Builds on

38 of 47 references.

References · 38

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
26 more
  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  4. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  6. 2022A Generalist Agentcited by 802
  7. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  8. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  9. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  10. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  11. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  12. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  13. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  14. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  15. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  16. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  17. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
  18. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  19. 2021Direct speech-to-speech translation with discrete unitscited by 148
  20. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  21. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  22. 2021Scaling Laws for Neural Machine Translationcited by 110
  23. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  24. 2022Causes and Cures for Interference in Multilingual Translationcited by 23
  25. 2021Scaling Laws for Acoustic Modelscited by 20
  26. 2022BARTSmiles: Generative Masked Language Models for Molecular Representationscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 30; 2024: 48; 2025: 35; 2026: 18.

131 papers.

Sort:

By area:

  1. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  2. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  3. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  4. 2023The Falcon Series of Open Language Modelscited by 526
  5. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  6. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  7. 2023Scaling Data-Constrained Language Modelscited by 391
  8. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  9. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  10. 2023Data Filtering Networkscited by 247
  11. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  12. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
38 more
  1. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  2. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  3. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  4. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
  5. 20234M: Massively Multimodal Masked Modelingcited by 118
  6. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  7. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
  8. 2024Scaling Laws for Precisioncited by 92
  9. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  10. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  11. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  12. 2023GIVT: Generative Infinite-Vocabulary Transformerscited by 80
  13. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 71
  14. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65
  15. 2023Goal Driven Discovery of Distributional Differences via Language Descriptionscited by 61
  16. 2023Goal-Driven Explainable Clustering via Language Descriptionscited by 58
  17. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  18. 2024Scaling Wearable Foundation Modelscited by 57
  19. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  20. 2025Scaling Laws for Optimal Data Mixturescited by 49
  21. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  22. 2025Scaling Laws for Native Multimodal Modelscited by 47
  23. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  24. 2024JetFormer: An Autoregressive Generative Model of Raw Images and Textcited by 44
  25. 2023OWL: A Large Language Model for IT Operationscited by 43
  26. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 40
  27. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39
  28. 2024Autoregressive Models in Vision: A Surveycited by 39
  29. 2024Core Knowledge Deficits in Multi-Modal Language Modelscited by 38
  30. 2025QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Trainingcited by 38
  31. 2024On the Scalability of GNNs for Molecular Graphscited by 36
  32. 2024Scaling Optimal LR Across Token Horizonscited by 35
  33. 2025Scaling Laws for Forgetting during Finetuning with Pretraining Data Injectioncited by 35
  34. 2024Foundation Models for Music: A Surveycited by 34
  35. 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33
  36. 2023Jointly Training Large Autoregressive Multimodal Modelscited by 32
  37. 2024Scaling Properties of Speech Language Modelscited by 30
  38. 2023Toward Joint Language Modeling for Speech Units and Textcited by 27

50 most cited of 131 All 131 (JSON)