mlsanityData to 7 Oct 2026

Jointly Training Large Autoregressive Multimodal Models

arXiv 2309.15564 ↗First posted 27 Sep 2023cs.LG cs.CL cs.CVcited by 32

Not on the map. Why?

Abstract (arXiv)

In recent years, advances in the large-scale pretraining of language and text-to-image models have revolutionized the field of machine learning. Yet, integrating these two modalities into a single, robust model capable of generating seamless multimodal outputs remains a significant challenge. To address this gap, we present the Joint Autoregressive Mixture (JAM) framework, a modular approach that systematically fuses existing text and image generation models. We also introduce a specialized, data-efficient instruction-tuning strategy, tailored for mixed-modal generation tasks. Our final instruct-tuned model demonstrates unparalleled performance in generating high-quality multimodal outputs and represents the first model explicitly designed for this purpose.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 45 of its 58 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

45 of 58 references.

References · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
33 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  13. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  14. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  15. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  16. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  17. 2023LIMA: Less Is More for Alignmentcited by 1,080
  18. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  19. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  20. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  21. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  22. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  23. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  24. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  25. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  26. 2023Generating Images with Multimodal Language Modelscited by 259
  27. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
  28. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  29. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  30. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  31. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  32. 2023A Theory on Adam Instability in Large-Scale Machine Learningcited by 54
  33. 2023Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learningcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 14; 2025: 8; 2026: 3.

32 papers.

Sort:

By area:

  1. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  2. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  3. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  4. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  5. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  6. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  7. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  8. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  9. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  10. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  11. 2024SEED-Story: Multimodal Long Story Generation with Large Language Modelcited by 70
  12. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
20 more
  1. 2024Grounding Multimodal Large Language Models in Actionscited by 35
  2. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  3. 2023LLMGA: Multimodal Large Language Model based Generation Assistantcited by 25
  4. 2025Generative AI for Cel-Animation: A Surveycited by 17
  5. 2025Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generationcited by 16
  6. 2023MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokenscited by 15
  7. 2023Expedited Training of Visual Conditioned Language Generation via Redundancy Reductioncited by 15
  8. 2024RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Datacited by 15
  9. 2024On Pre-training of Multimodal Language Models Customized for Chart Understandingcited by 13
  10. 2023Generating Illustrated Instructionscited by 7
  11. 2025DoraCycle: Domain-Oriented Adaptation of Unified Generative Model in Multimodal Cyclescited by 5
  12. 2026When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Mergingcited by 5
  13. 2024DoLLM: How Large Language Models Understanding Network Flow Data to Detect Carpet Bombing DDoScited by 4
  14. 2024MUMU: Bootstrapping Multimodal Image Generation from Text-to-Image Datacited by 4
  15. 2024Olympus: A Universal Task Router for Computer Vision Taskscited by 4
  16. 2025UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depthscited by 4
  17. 2025Harmonizing and Merging Source Models for CLIP-based Domain Generalizationcited by 2
  18. 2026MMGRid: Navigating Temporal-aware and Cross-domain Generative Recommendation via Model Mergingcited by 1
  19. 2023On the Evaluation and Refinement of Vision-Language Instruction Tuning Datasetscited by 0
  20. 2026Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transportcited by 0