mlsanityData to 7 Oct 2026

Visual Prompting via Image Inpainting

arXiv 2209.00647 ↗First posted 1 Sep 2022cs.CVcited by 248

Not on the map. Why?

Cited by 31 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

How does one adapt a pre-trained visual model to novel downstream tasks without task-specific finetuning or any model modification? Inspired by prompting in NLP, this paper investigates visual prompting: given input-output image example(s) of a new task at test time and a new input image, the goal is to automatically produce the output image, consistent with the given examples. We show that posing this problem as simple image inpainting - literally just filling in a hole in a concatenated visual prompt image - turns out to be surprisingly effective, provided that the inpainting algorithm has been trained on the right data. We train masked auto-encoders on a new dataset that we curated - 88k unlabeled figures from academic papers sources on Arxiv. We apply visual prompting to these pretrained models and demonstrate results on various downstream image-to-image tasks, including foreground segmentation, single object detection, colorization, edge detection, etc.

Key facts. Cited by 248 papers in the map (data through 7 Oct 2026). Builds on 44 of its 61 references that are in the map. First posted on arXiv 1 Sep 2022.

Builds on

44 of 61 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
32 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  5. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  8. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  9. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  11. 2017Deep Image Priorcited by 1,097
  12. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  13. 2019How Can We Know What Language Models Know?cited by 953
  14. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  15. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  16. 2018Image Inpainting for Irregular Holes Using Partial Convolutionscited by 505
  17. 2017One-Shot Learning for Semantic Segmentationcited by 337
  18. 2020Frustratingly Simple Few-Shot Object Detectioncited by 280
  19. 2020Prior Guided Feature Enrichment Network for Few-Shot Segmentationcited by 271
  20. 2018Few-shot Object Detection via Feature Reweightingcited by 265
  21. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  22. 2020Prototype Mixture Models for Few-shot Semantic Segmentationcited by 190
  23. 2019Feature Weighting and Boosting for Few-Shot Segmentationcited by 189
  24. 2020Part-aware Prototype Network for Few-shot Semantic Segmentationcited by 162
  25. 2021Few-Shot Segmentation via Cycle-Consistent Transformercited by 133
  26. 2016High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesiscited by 89
  27. 2021Diverse Image Inpainting with Bidirectional and Autoregressive Transformerscited by 80
  28. 2021DETReg: Unsupervised Pretraining with Region Priors for Object Detectioncited by 76
  29. 2020Learning Representations by Predicting Bags of Visual Wordscited by 73
  30. 2019Learning to Make Analogies by Contrasting Abstract Relational Structurecited by 53
  31. 2015VISALOGY: Answering Visual Analogy Questionscited by 17
  32. 2016From A to Z: Supervised Transfer of Style and Content Using Deep Neural Network Generatorscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 64; 2024: 85; 2025: 55; 2026: 38.

248 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  3. 2022A Survey on In-context Learningcited by 717
  4. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  5. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  6. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  7. 2023Personalize Segment Anything Model with One Shotcited by 250
  8. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  9. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  10. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 197
  11. 2023SegGPT: Segmenting Everything In Contextcited by 193
  12. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162
38 more
  1. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  2. 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 140
  3. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  4. 20234M: Massively Multimodal Masked Modelingcited by 118
  5. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Modelscited by 117
  6. 2023Explicit Visual Prompting for Low-Level Structure Segmentationscited by 113
  7. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  8. 2024Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Promptscited by 88
  9. 2022Understanding and Improving Visual Prompting: A Label-Mapping Perspectivecited by 86
  10. 2024Video as the New Language for Real-World Decision Makingcited by 86
  11. 2023Fine-Grained Visual Promptingcited by 81
  12. 2023ProRes: Exploring Degradation-aware Visual Prompt for Universal Image Restorationcited by 77
  13. 2024VRP-SAM: SAM with Visual Reference Promptcited by 75
  14. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 74
  15. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 73
  16. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  17. 2023NeRFiller: Completing Scenes via Generative 3D Inpaintingcited by 62
  18. 2024Perceive-IR: Learning to Perceive Degradation Better for All-in-One Image Restorationcited by 60
  19. 2023ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulationcited by 59
  20. 2024A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trendscited by 58
  21. 2023Putting People in Their Place: Affordance-Aware Human Insertion into Scenescited by 53
  22. 2022Multitask Vision-Language Prompt Tuningcited by 51
  23. 2024Towards Effective Multiple-in-One Image Restoration: A Sequential and Prompt Learning Strategycited by 48
  24. 2024ARC: A Generalist Graph Anomaly Detector with In-Context Learningcited by 46
  25. 2023Towards In-context Scene Understandingcited by 45
  26. 2023StyleGAN knows Normal, Depth, Albedo, and Morecited by 44
  27. 2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43
  28. 2023MetaCloak: Preventing Unauthorized Subject-driven Text-to-image Diffusion-based Synthesis via Meta-learningcited by 42
  29. 2023Dual-path Adaptation from Image to Video Transformerscited by 40
  30. 2023Explore In-Context Learning for 3D Point Cloud Understandingcited by 40
  31. 2023From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videoscited by 39
  32. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Modelscited by 39
  33. 2024Autoregressive Models in Vision: A Surveycited by 39
  34. 2025VisualCloze: A Universal Image Generation Framework via Visual In-Context Learningcited by 39
  35. 2024Customizing Text-to-Image Models with a Single Image Paircited by 36
  36. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  37. 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 34
  38. 2023EventBind: Learning a Unified Representation to Bind Them All for Event-based Open-world Understandingcited by 33

50 most cited of 248 All 248 (JSON)