mlsanityData to 13 Oct 2026

A Unified Sequence Interface for Vision Tasks

arXiv 2206.07669 ↗First posted 15 Jun 2022cs.CV cs.CL cs.LGcited by 152

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While language tasks are naturally expressed in a single, unified, modeling framework, i.e., generating sequences of tokens, this has not been the case in computer vision. As a result, there is a proliferation of distinct architectures and loss functions for different vision tasks. In this work we show that a diverse set of "core" computer vision tasks can also be unified if formulated in terms of a shared pixel-to-sequence interface. We focus on four tasks, namely, object detection, instance segmentation, keypoint detection, and image captioning, all with diverse types of outputs, e.g., bounding boxes or dense masks. Despite that, by formulating the output of each task as a sequence of discrete tokens with a unified interface, we show that one can train a neural network with a single model architecture and loss function on all these tasks, with no task-specific customization. To solve a specific task, we use a short prompt as task description, and the sequence output adapts to the prompt so it can produce task-specific output. We show that such a model can achieve competitive performance compared to well-established task-specific models.

Key facts. Cited by 152 papers in the map (data through 13 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 15 Jun 2022.

Builds on

33 of 56 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,656
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,625
  11. 2020End-to-End Object Detection with Transformerscited by 7,396
  12. 2017Mask R-CNNcited by 7,266
21 more
  1. 2016Feature Pyramid Networks for Object Detectioncited by 5,027
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  3. 2015Fast R-CNNcited by 2,933
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,893
  5. 2017Non-local Neural Networkscited by 2,291
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,452
  7. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,150
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,094
  9. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,062
  10. 2022A Generalist Agentcited by 803
  11. 2019Big Transfer (BiT): General Visual Representation Learningcited by 768
  12. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  13. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 625
  14. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  15. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  16. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  17. 2020X-Linear Attention Networks for Image Captioningcited by 198
  18. 2021Benchmarking Detection Transfer Learning with Vision Transformerscited by 130
  19. 2017Annotating Object Instances with a Polygon-RNNcited by 88
  20. 2021Multi-Task Self-Training for Learning General Representationscited by 85
  21. 2022UViM: A Unified Modeling Approach for Vision with Learned Guiding Codescited by 77

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 13; 2023: 53; 2024: 43; 2025: 25; 2026: 18.

152 papers.

Sort:

By area:

  1. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 764
  2. 2023Segment Everything Everywhere All at Oncecited by 547
  3. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 503
  4. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  5. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  6. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 406
  7. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 404
  8. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 365
  9. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 362
  10. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 335
  11. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  12. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
38 more
  1. 2022Generalized Decoding for Pixel, Image, and Languagecited by 293
  2. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  3. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 247
  4. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  5. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  6. 2023SegGPT: Segmenting Everything In Contextcited by 193
  7. 2022Scalable Adaptive Computation for Iterative Generationcited by 192
  8. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 191
  9. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 163
  10. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  11. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  12. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 136
  13. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  14. 20234M: Massively Multimodal Masked Modelingcited by 120
  15. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  16. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  17. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  18. 2022A Generalist Framework for Panoptic Segmentation of Images and Videoscited by 99
  19. 2024Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Promptscited by 89
  20. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 74
  21. 2023General Object Foundation Model for Images and Videos at Scalecited by 71
  22. 2023UniHCP: A Unified Model for Human-Centric Perceptionscited by 59
  23. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  24. 2024A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trendscited by 59
  25. 2022Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Taskscited by 57
  26. 2025Uni-Sign: Toward Unified Sign Language Understanding at Scalecited by 56
  27. 2023All in Tokens: Unifying Output Space of Visual Tasks via Soft Tokencited by 52
  28. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 52
  29. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  30. 2023LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editingcited by 47
  31. 2024ARC: A Generalist Graph Anomaly Detector with In-Context Learningcited by 46
  32. 2023Human Pose as Compositional Tokenscited by 45
  33. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 45
  34. 2024HecVL: Hierarchical Video-Language Pretraining for Zero-shot Surgical Phase Recognitioncited by 41
  35. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perceptioncited by 39
  36. 2025RemoteSAM: Towards Segment Anything for Earth Observationcited by 39
  37. 2024M3GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generationcited by 38
  38. 2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31

50 most cited of 152 All 152 (JSON)