mlsanityData to 7 Oct 2026

Images Speak in Images: A Generalist Painter for In-Context Visual Learning

arXiv 2212.02499 ↗First posted 5 Dec 2022cs.CVcited by 317

Not on the map. Why?

Cited by 38 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In-context learning, as a new paradigm in NLP, allows the model to rapidly adapt to various tasks with only a handful of prompts and examples. But in computer vision, the difficulties for in-context learning lie in that tasks vary significantly in the output representations, thus it is unclear how to define the general-purpose task prompts that the vision model can understand and transfer to out-of-domain tasks. In this work, we present Painter, a generalist model which addresses these obstacles with an "image"-centric solution, that is, to redefine the output of core vision tasks as images, and specify task prompts as also images. With this idea, our training process is extremely simple, which performs standard masked image modeling on the stitch of input and output image pairs. This makes the model capable of performing tasks conditioned on visible image patches. Thus, during inference, we can adopt a pair of input and output images from the same task as the input condition, to indicate which task to perform. Without bells and whistles, our generalist Painter can achieve competitive performance compared to well-established task-specific models, on seven representative vision tasks ranging from high-level visual understanding to low-level image processing. In addition, Painter significantly outperforms recent generalist models on several challenging tasks.

Key facts. Cited by 317 papers in the map (data through 7 Oct 2026). Builds on 49 of its 58 references that are in the map. First posted on arXiv 5 Dec 2022.

Builds on

49 of 58 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
37 more
  1. 2015Fast R-CNNcited by 2,930
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  7. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  8. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  9. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  10. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  11. 2016Deep Networks with Stochastic Depthcited by 973
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  13. 2021Uformer: A General U-Shaped Transformer for Image Restorationcited by 864
  14. 2021Multi-Stage Progressive Image Restorationcited by 813
  15. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  16. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  17. 2018Deep Retinex Decomposition for Low-Light Enhancementcited by 720
  18. 2019Panoptic Feature Pyramid Networkscited by 644
  19. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  20. 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
  21. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  22. 2016Deep Joint Rain Detection and Removal from a Single Imagecited by 351
  23. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  24. 2019SOLO: Segmenting Objects by Locationscited by 289
  25. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  26. 2018Density-aware Single Image De-raining using a Multi-stream Dense Networkcited by 248
  27. 2022Visual Prompting via Image Inpaintingcited by 248
  28. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  29. 2017Image De-raining Using a Conditional Generative Adversarial Networkcited by 197
  30. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  31. 2022BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimationcited by 166
  32. 2022A Unified Sequence Interface for Vision Taskscited by 152
  33. 2022Learning Enriched Features for Fast Image Restoration and Enhancementcited by 150
  34. 2018High Quality Monocular Depth Estimation via Transfer Learningcited by 133
  35. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  36. 2022Language Models are General-Purpose Interfacescited by 96
  37. 2022UViM: A Unified Modeling Approach for Vision with Learned Guiding Codescited by 77

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 73; 2024: 114; 2025: 78; 2026: 51.

317 papers.

Sort:

By area:

  1. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  2. 2022A Survey on In-context Learningcited by 717
  3. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  4. 2023Segment Everything Everywhere All at Oncecited by 546
  5. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  6. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  7. 2024OmniGen: Unified Image Generationcited by 388
  8. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  9. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  10. 2023Personalize Segment Anything Model with One Shotcited by 250
  11. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  12. 2025Video models are zero-shot learners and reasonerscited by 208
38 more
  1. 2023SegGPT: Segmenting Everything In Contextcited by 193
  2. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  3. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162
  4. 2024Medical SAM 2: Segment medical images as video via Segment Anything Model 2cited by 151
  5. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  6. 2023Matcher: Segment Anything with One Shot Using All-Purpose Feature Matchingcited by 142
  7. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  8. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  9. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  10. 20234M: Massively Multimodal Masked Modelingcited by 118
  11. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  12. 2023Diffusion Models for Image Restoration and Enhancement: A Comprehensive Surveycited by 108
  13. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  14. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  15. 2023UniverSeg: Universal Medical Image Segmentationcited by 90
  16. 2024Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Promptscited by 88
  17. 2024Video as the New Language for Real-World Decision Makingcited by 86
  18. 2023Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Trainingcited by 85
  19. 2023Segment Anything Meets Point Trackingcited by 81
  20. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 77
  21. 2023ProRes: Exploring Degradation-aware Visual Prompt for Universal Image Restorationcited by 77
  22. 2024VRP-SAM: SAM with Visual Reference Promptcited by 75
  23. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
  24. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 74
  25. 2024What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?cited by 74
  26. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 73
  27. 2024Selective Hourglass Mapping for Universal Image Restoration Based on Diffusion Modelcited by 73
  28. 2023A Comprehensive Survey on Segment Anything Model for Vision and Beyondcited by 64
  29. 2023ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulationcited by 59
  30. 2024A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trendscited by 58
  31. 2023Visual In-Context Promptingcited by 52
  32. 2023LLaFS: When Large Language Models Meet Few-Shot Segmentationcited by 48
  33. 2023ScribblePrompt: Fast and Flexible Interactive Segmentation for Any Biomedical Imagecited by 47
  34. 2024UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Modelscited by 47
  35. 2024X-VILA: Cross-Modality Alignment for Large Language Modelcited by 46
  36. 2023Towards In-context Scene Understandingcited by 45
  37. 2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43
  38. 2025VideoWorld: Exploring Knowledge Learning from Unlabeled Videoscited by 42

50 most cited of 317 All 317 (JSON)