mlsanityData to 7 Oct 2026

UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes

arXiv 2205.10337 ↗First posted 20 May 2022cs.CVcited by 77

Not on the map. Why?

Abstract (arXiv)

We introduce UViM, a unified approach capable of modeling a wide range of computer vision tasks. In contrast to previous models, UViM has the same functional form for all tasks; it requires no task-specific modifications which require extensive human expertise. The approach involves two components: (I) a base model (feed-forward) which is trained to directly predict raw vision outputs, guided by a learned discrete code and (II) a language model (autoregressive) that is trained to generate the guiding code. These components complement each other: the language model is well-suited to modeling structured interdependent data, while the base model is efficient at dealing with high-dimensional outputs. We demonstrate the effectiveness of UViM on three diverse and challenging vision tasks: panoptic segmentation, depth prediction and image colorization, where we achieve competitive and near state-of-the-art results. Our experimental results suggest that UViM is a promising candidate for a unified modeling approach in computer vision.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 47 of its 62 references that are in the map. First posted on arXiv 20 May 2022.

Builds on

47 of 62 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  11. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  12. 2020End-to-End Object Detection with Transformerscited by 7,384
35 more
  1. 2017Mask R-CNNcited by 7,259
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2013Auto-Encoding Variational Bayescited by 3,980
  8. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  12. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  13. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  14. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  15. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  16. 2016Colorful Image Colorizationcited by 1,113
  17. 2021Scaling Vision Transformerscited by 993
  18. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  19. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  20. 2016Pixel Recurrent Neural Networkscited by 814
  21. 2018Panoptic Segmentationcited by 796
  22. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  23. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  24. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  25. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  26. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  27. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  28. 2022BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimationcited by 166
  29. 2018High Quality Monocular Depth Estimation via Transfer Learningcited by 133
  30. 2021Colorization Transformercited by 89
  31. 2022Transframer: Arbitrary Frame Prediction with Generative Modelscited by 37
  32. 2017PixColor: Pixel Recursive Colorizationcited by 32
  33. 2018Detecting Visual Relationships Using Box Attentioncited by 16
  34. 2016Chained Predictions Using Convolutional Neural Networkscited by 15
  35. 2017Probabilistic Image Colorizationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 24; 2024: 21; 2025: 13; 2026: 7.

77 papers.

Sort:

By area:

  1. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  2. 2023Segment Everything Everywhere All at Oncecited by 546
  3. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  4. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  5. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  6. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  7. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  8. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  9. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  10. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  11. 2023SegGPT: Segmenting Everything In Contextcited by 193
  12. 2022A Unified Sequence Interface for Vision Taskscited by 152
38 more
  1. 2022FlexiViT: One Model for All Patch Sizescited by 151
  2. 20234M: Massively Multimodal Masked Modelingcited by 118
  3. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  4. 2022A Generalist Framework for Panoptic Segmentation of Images and Videoscited by 99
  5. 2024Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Promptscited by 88
  6. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 87
  7. 2023GIVT: Generative Infinite-Vocabulary Transformerscited by 80
  8. 2023UniHCP: A Unified Model for Human-Centric Perceptionscited by 59
  9. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 53
  10. 2023All in Tokens: Unifying Output Space of Visual Tasks via Soft Tokencited by 52
  11. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  12. 2024Restructuring Vector Quantization with the Rotation Trickcited by 49
  13. 2022Retrospectives on the Embodied AI Workshopcited by 46
  14. 2024ARC: A Generalist Graph Anomaly Detector with In-Context Learningcited by 46
  15. 2024JetFormer: An Autoregressive Generative Model of Raw Images and Textcited by 44
  16. 2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43
  17. 2022RUST: Latent Neural Scene Representations from Unposed Imagerycited by 41
  18. 2023Tuning computer vision models with task rewardscited by 36
  19. 2023Neural Graph Reasoning: Complex Logical Query Answering Meets Graph Databasescited by 35
  20. 2023TarViS: A Unified Approach for Target-based Video Segmentationcited by 32
  21. 2023Generative Semantic Segmentationcited by 32
  22. 2023InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalistscited by 30
  23. 2023UniRef++: Segment Every Reference Object in Spatial and Temporal Spacescited by 22
  24. 2024LaVin-DiT: Large Vision Diffusion Transformercited by 21
  25. 2022Egocentric Video Task Translationcited by 19
  26. 2023Selective Visual Representations Improve Convergence and Generalization for Embodied AIcited by 19
  27. 2024A Simple Image Segmentation Framework via In-Context Examplescited by 19
  28. 2024A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpaintingcited by 18
  29. 2025ARIG: Autoregressive Interactive Head Generation for Real-time Conversationscited by 16
  30. 2023PolyMaX: General Dense Prediction with Mask Transformercited by 14
  31. 2024Data-efficient Large Vision Models through Sequential Autoregressioncited by 14
  32. 2026OAT: Ordered Action Tokenizationcited by 14
  33. 2024Denoising with a Joint-Embedding Predictive Architecturecited by 12
  34. 2025OmniCast: A Masked Latent Diffusion Model for Weather Forecasting Across Time Scalescited by 11
  35. 2023A Study of Autoregressive Decoders for Multi-Tasking in Computer Visioncited by 10
  36. 2025EgoM2P: Egocentric Multimodal Multitask Pretrainingcited by 9
  37. 2024Chameleon: A Data-Efficient Generalist for Dense Visual Prediction in the Wildcited by 6
  38. 2023Towards AGI in Computer Vision: Lessons Learned from GPT and Large Language Modelscited by 5

50 most cited of 77 All 77 (JSON)