# Stable Diffusion Models are Secretly Good at Visual In-Context Learning arXiv: https://arxiv.org/abs/2508.09949 ยท HTML: https://mlsanity.com/paper/2508.09949/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. - Builds on 44 of its 48 references that are in the map. - First posted on arXiv 13 Aug 2025. ## Placed in - Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/) ## Builds on (most cited first, 30 of 44) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014) - [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021) - [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022) - [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022) - [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016) - [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022) - [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020) - [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022) - [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022) - [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017) - [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022) - [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021) - [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016) - [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022) - [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023) - [Improved Distribution Matching Distillation for Fast Image Synthesis](https://mlsanity.com/paper/2405.14867/) (2024) - [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015) - [InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation](https://mlsanity.com/paper/2309.06380/) (2023) - [One-Shot Learning for Semantic Segmentation](https://mlsanity.com/paper/1709.03410/) (2017) - [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](https://mlsanity.com/paper/2212.02499/) (2022) - [Visual Prompting via Image Inpainting](https://mlsanity.com/paper/2209.00647/) (2022) - [An Edit Friendly DDPM Noise Space: Inversion and Manipulations](https://mlsanity.com/paper/2304.06140/) (2023) ## Built on by (most cited first, 8 of 8) - [Splatent: Splatting Diffusion Latents for Novel View Synthesis](https://mlsanity.com/paper/2512.09923/) (2025) - [Stability of In-Context Learning: A Spectral Coverage Perspective](https://mlsanity.com/paper/2509.20677/) (2025) - [PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignment](https://mlsanity.com/paper/2603.18891/) (2026) - [The Latent Color Subspace: Emergent Order in High-Dimensional Chaos](https://mlsanity.com/paper/2603.12261/) (2026) - [Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning](https://mlsanity.com/paper/2604.03657/) (2026) - [Beyond Text Prompts: Visual-to-Visual Generation as A Unified Paradigm](https://mlsanity.com/paper/2605.12271/) (2026) - [Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Tasks](https://mlsanity.com/paper/2606.10967/) (2026) - [Hidden-Shot: Towards One-Shot Task Generalization for Low-Level Vision Generalist Models](https://mlsanity.com/paper/2607.01535/) (2026) ## Abstract (arXiv) Large language models (LLM) in natural language processing (NLP) have demonstrated great potential for in-context learning (ICL) -- the ability to leverage a few sets of example prompts to adapt to various tasks without having to explicitly update the model weights. ICL has recently been explored for computer vision tasks with promising early outcomes. These approaches involve specialized training and/or additional data that complicate the process and limit its generalizability. In this work, we show that off-the-shelf Stable Diffusion models can be repurposed for visual in-context learning (V-ICL). Specifically, we formulate an in-place attention re-computation within the self-attention layers of the Stable Diffusion architecture that explicitly incorporates context between the query and example prompts. Without any additional fine-tuning, we show that this repurposed Stable Diffusion model is able to adapt to six different tasks: foreground segmentation, single object detection, semantic segmentation, keypoint detection, edge detection, and colorization. For example, the proposed approach improves the mean intersection over union (mIoU) for the foreground segmentation task on Pascal-5i dataset by 8.9% and 3.2% over recent methods such as Visual Prompting and IMProv, respectively. Additionally, we show that the proposed method is able to effectively leverage multiple prompts through ensembling to infer the task better and further improve the performance.