mlsanityData to 7 Oct 2026

Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action

arXiv 2312.17172 ↗First posted 28 Dec 2023cs.CV cs.AI cs.CLcited by 301

Not on the map. Why?

Cited by 75 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Unified-IO 2, the first autoregressive multimodal model that is capable of understanding and generating image, text, audio, and action. To unify different modalities, we tokenize inputs and outputs -- images, text, audio, action, bounding boxes, etc., into a shared semantic space and then process them with a single encoder-decoder transformer model. Since training with such diverse modalities is challenging, we propose various architectural improvements to stabilize model training. We train our model from scratch on a large multimodal pre-training corpus from diverse sources with a multimodal mixture of denoisers objective. To learn an expansive set of skills, such as following multimodal instructions, we construct and finetune on an ensemble of 120 datasets with prompts and augmentations. With a single unified model, Unified-IO 2 achieves state-of-the-art performance on the GRIT benchmark and strong results in more than 35 benchmarks, including image generation and understanding, natural language understanding, video and audio understanding, and robotic manipulation. We release all our models to the research community.

Key facts. Cited by 301 papers in the map (data through 7 Oct 2026). Builds on 184 of its 226 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

184 of 226 references.

References · 184

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Segment Anythingcited by 8,158
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  12. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  2. 2017Mask R-CNNcited by 7,259
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2016Layer Normalizationcited by 5,579
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  13. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  14. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  15. 2021Program Synthesis with Large Language Modelscited by 3,509
  16. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  17. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  18. 2016Improved Techniques for Training GANscited by 2,938
  19. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  20. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  21. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  22. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  23. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  24. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  25. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  26. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  27. 2015VQA: Visual Question Answeringcited by 2,489
  28. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  29. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  30. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  31. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  32. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  33. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  34. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  35. 2019Towards VQA Models That Can Readcited by 1,696
  36. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  37. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  38. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530

50 most cited of 184 All 184 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 103; 2025: 123; 2026: 74.

301 papers.

Sort:

By area:

  1. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  2. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  3. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  4. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  5. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  6. 2025Qwen2.5-Omni Technical Reportcited by 704
  7. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  8. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  9. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  10. 2024OmniGen: Unified Image Generationcited by 388
  11. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  12. 2024What matters when building vision-language models?cited by 321
38 more
  1. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  2. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  3. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  4. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  5. 2024Rotary Position Embedding for Vision Transformercited by 226
  6. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  7. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  8. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  9. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  10. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  11. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  12. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  13. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  14. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  15. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  16. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  17. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 112
  18. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  19. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  20. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  21. 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 94
  22. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  23. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  24. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  25. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  26. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 68
  27. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Modelscited by 68
  28. 2024ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhancecited by 64
  29. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 62
  30. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 61
  31. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Modelcited by 60
  32. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinementcited by 53
  33. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 53
  34. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  35. 2025UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encodingcited by 49
  36. 2025VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulationcited by 48
  37. 2024Interleaved-Modal Chain-of-Thoughtcited by 47
  38. 2024X-VILA: Cross-Modality Alignment for Large Language Modelcited by 46

50 most cited of 301 All 301 (JSON)