mlsanityData to 7 Oct 2026

In-Context Learning Unlocked for Diffusion Models

arXiv 2305.01115 ↗First posted 1 May 2023cs.CVcited by 103

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

We present Prompt Diffusion, a framework for enabling in-context learning in diffusion-based generative models. Given a pair of task-specific example images, such as depth from/to image and scribble from/to image, and a text guidance, our model automatically understands the underlying task and performs the same task on a new query image following the text guidance. To achieve this, we propose a vision-language prompt that can model a wide range of vision-language tasks and a diffusion model that takes it as input. The diffusion model is trained jointly over six different tasks using these prompts. The resulting Prompt Diffusion model is the first diffusion-based vision-language foundation model capable of in-context learning. It demonstrates high-quality in-context generation on the trained tasks and generalizes effectively to new, unseen vision tasks with their respective prompts. Our model also shows compelling text-guided image editing results. Our framework aims to facilitate research into in-context learning for computer vision. We share our code and pre-trained models at https://github.com/Zhendong-Wang/Prompt-Diffusion.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 65 of its 75 references that are in the map. First posted on arXiv 1 May 2023.

Builds on

65 of 75 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyond11 Apr
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  4. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  5. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Model15 Nov
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  7. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  13. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  14. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  15. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  16. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  17. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  18. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  19. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  20. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  21. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  22. 2015Holistically-Nested Edge Detectioncited by 590
  23. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  24. 2023Zero-shot Image-to-Image Translationcited by 503
  25. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  26. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  27. 2022Visual Prompting via Image Inpaintingcited by 248
  28. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  29. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  30. 2022Diffusion-GAN: Training GANs with Diffusioncited by 203
  31. 2023SegGPT: Segmenting Everything In Contextcited by 193
  32. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  33. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  34. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  35. 2021On Fast Sampling of Diffusion Probabilistic Modelscited by 170
  36. 2022A Unified Sequence Interface for Vision Taskscited by 152
  37. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  38. 2022NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360deg Viewscited by 148

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 36; 2025: 26; 2026: 23.

103 papers.

Sort:

By area:

  1. 2023To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Nowcited by 204
  2. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 197
  3. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162
  4. 2024StableNormal: Reducing Diffusion Variance for Stable and Sharp Normalcited by 153
  5. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  6. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  7. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  8. 2024Multi-LoRA Composition for Image Generationcited by 80
  9. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 77
  10. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  11. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  12. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generationcited by 63
38 more
  1. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Modelscited by 62
  2. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 52
  3. 2023Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Modelscited by 48
  4. 2024UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Modelscited by 47
  5. 2025VisualCloze: A Universal Image Generation Framework via Visual In-Context Learningcited by 39
  6. 2023Exploiting Diffusion Prior for Generalizable Dense Predictioncited by 37
  7. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  8. 2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
  9. 2024Analogist: Out-of-the-box Visual In-Context Learning with Image Diffusion Modelcited by 30
  10. 2024Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimizationcited by 30
  11. 2025X-SAM: From Segment Anything to Any Segmentationcited by 30
  12. 2023Visual Instruction Inversion: Image Editing via Visual Promptingcited by 26
  13. 2024DiffClass: Diffusion-Based Class Incremental Learningcited by 23
  14. 2024Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Modelscited by 22
  15. 2024LaVin-DiT: Large Vision Diffusion Transformercited by 21
  16. 2023Context Diffusion: In-Context Aware Image Generationcited by 20
  17. 2024Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Datacited by 20
  18. 2025ImageRAG: Dynamic Image Retrieval for Reference-Guided Image Generationcited by 20
  19. 2023Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Modelscited by 18
  20. 2024A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpaintingcited by 18
  21. 2023Beta Diffusioncited by 17
  22. 2024Towards Global Optimal Visual In-Context Learning Prompt Selectioncited by 15
  23. 2023InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generationcited by 14
  24. 2025Lumina-OmniLV: A Unified Multimodal Framework for General Low-Level Visioncited by 14
  25. 2024Low-Resource Vision Challenges for Foundation Modelscited by 13
  26. 2024Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulationcited by 13
  27. 2023MoEController: Instruction-based Arbitrary Image Manipulation with Mixture-of-Expert Controllerscited by 12
  28. 2024Explore In-Context Segmentation via Latent Diffusion Modelscited by 12
  29. 2024InstructGIE: Towards Generalizable Image Editingcited by 11
  30. 2024InstructBrush: Learning Attention-based Instruction Optimization for Image Editingcited by 10
  31. 2024Guided Score identity Distillation for Data-Free One-Step Text-to-Image Generationcited by 10
  32. 2023Improving In-Context Learning in Diffusion Models with Visual Context-Modulated Promptscited by 9
  33. 2025Can World Models Benefit VLMs for World Dynamics?cited by 9
  34. 2024Disrupting Diffusion-based Inpainters with Semantic Digressioncited by 8
  35. 2025PairEdit: Learning Semantic Variations for Exemplar-based Image Editingcited by 8
  36. 2025Stable Diffusion Models are Secretly Good at Visual In-Context Learningcited by 8
  37. 2025Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learningcited by 7
  38. 2025In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulationcited by 7

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Harnessing Diffusion Models for Visual Perception with Meta Prompts
  2. 2023InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists
  3. 2024X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Models
  4. 2023Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models
  5. 2024Prompt Diffusion Robustifies Any-Modality Prompt Learning
  6. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models
  7. 2024Dual Diffusion for Unified Image Generation and Understanding
  8. 2023Unleashing Text-to-Image Diffusion Models for Visual Perception