mlsanityData to 14 Oct 2026

Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks

arXiv 2206.08916 ↗First posted 17 Jun 2022cs.CVcited by 421

Not on the map. Why?

Cited by 94 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose Unified-IO, a model that performs a large variety of AI tasks spanning classical computer vision tasks, including pose estimation, object detection, depth estimation and image generation, vision-and-language tasks such as region captioning and referring expression, to natural language processing tasks such as question answering and paraphrasing. Developing a single unified model for such a large variety of tasks poses unique challenges due to the heterogeneous inputs and outputs pertaining to each task, including RGB images, per-pixel maps, binary masks, bounding boxes, and language. We achieve this unification by homogenizing every supported input and output into a sequence of discrete vocabulary tokens. This common representation across all tasks allows us to train a single transformer-based architecture, jointly on over 90 diverse datasets in the vision and language fields. Unified-IO is the first model capable of performing all 7 tasks on the GRIT benchmark and produces strong results across 16 diverse benchmarks like NYUv2-Depth, ImageNet, VQA2.0, OK-VQA, Swig, VizWizGround, BoolQ, and SciTail, with no task-specific fine-tuning. Code and demos for Unified-IO are available at: https://unified-io.allenai.org.

Key facts. Cited by 421 papers in the map (data through 14 Oct 2026). Builds on 92 of its 129 references that are in the map. First posted on arXiv 17 Jun 2022.

Builds on

92 of 129 references.

References · 92

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2017Mask R-CNNcited by 7,277
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2015VQA: Visual Question Answeringcited by 2,506
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  13. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  14. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  15. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  16. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  17. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  18. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  19. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  20. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  21. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  22. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  23. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  24. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  25. 2021Grounded Language-Image Pre-trainingcited by 1,156
  26. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  27. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  28. 2018Neural Network Acceptability Judgmentscited by 958
  29. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  30. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  31. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  32. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  33. 2022A Generalist Agentcited by 808
  34. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  35. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  36. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  37. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  38. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 27; 2023: 143; 2024: 122; 2025: 89; 2026: 40.

421 papers.

Sort:

By area:

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  3. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  4. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  5. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  6. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  7. 2023Segment Everything Everywhere All at Oncecited by 548
  8. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  9. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  10. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  11. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  12. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
38 more
  1. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  2. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  3. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  4. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 339
  5. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 328
  6. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  7. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 315
  8. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  9. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  10. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  11. 2023Towards Generalist Biomedical AIcited by 289
  12. 2023Generating Images with Multimodal Language Modelscited by 260
  13. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  14. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  15. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  16. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  17. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 237
  18. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  19. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 222
  20. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  21. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  22. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 205
  23. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 203
  24. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
  25. 2023SegGPT: Segmenting Everything In Contextcited by 194
  26. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 192
  27. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 166
  28. 2023Language-Driven Representation Learning for Roboticscited by 165
  29. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 164
  30. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  31. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  32. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 150
  33. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 148
  34. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  35. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  36. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  37. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 133
  38. 2023Transformer-Based Visual Segmentation: A Surveycited by 124

50 most cited of 421 All 421 (JSON)