# Context Diffusion: In-Context Aware Image Generation

arXiv: https://arxiv.org/abs/2312.03584 · HTML: https://mlsanity.com/paper/2312.03584/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 57 references that are in the map.
- First posted on arXiv 6 Dec 2023.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)

## Built on by (most cited first, 21 of 21)

- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [A Survey on Personalized Content Synthesis with Diffusion Models](https://mlsanity.com/paper/2405.05538/) (2024)
- [ImageRAG: Dynamic Image Retrieval for Reference-Guided Image Generation](https://mlsanity.com/paper/2502.09411/) (2025)
- [VINCIE: Unlocking In-context Image Editing from Video](https://mlsanity.com/paper/2506.10941/) (2025)
- [XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning](https://mlsanity.com/paper/2406.08973/) (2024)
- [Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning](https://mlsanity.com/paper/2504.21263/) (2025)
- [In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulation](https://mlsanity.com/paper/2505.20271/) (2025)
- [Pointmap-Conditioned Diffusion for Consistent Novel View Synthesis](https://mlsanity.com/paper/2501.02913/) (2025)
- [Does Feasibility Matter? Understanding the Impact of Feasibility on Synthetic Training Data](https://mlsanity.com/paper/2505.10551/) (2025)
- [INRetouch: Context Aware Implicit Neural Representation for Photography Retouching](https://mlsanity.com/paper/2412.03848/) (2024)
- [IA-T2I: Internet-Augmented Text-to-Image Generation](https://mlsanity.com/paper/2505.15779/) (2025)
- [In-Context Sync-LoRA for Portrait Video Editing](https://mlsanity.com/paper/2512.03013/) (2025)
- [Conditional Diffusion on Web-Scale Image Pairs leads to Diverse Image Variations](https://mlsanity.com/paper/2405.14857/) (2024)
- [Vision-Driven Prompt Optimization for Large Language Models in Multimodal Generative Tasks](https://mlsanity.com/paper/2501.02527/) (2025)
- [Underlying Semantic Diffusion for Effective and Efficient In-Context Learning](https://mlsanity.com/paper/2503.04050/) (2025)
- [Sissi: Zero-shot Style-guided Image Synthesis via Semantic-style Integration](https://mlsanity.com/paper/2601.06605/) (2026)
- [OPRO: Orthogonal Panel-Relative Operators for Panel-Aware In-Context Image Generation](https://mlsanity.com/paper/2603.27637/) (2026)
- [Beyond Text Prompts: Visual-to-Visual Generation as A Unified Paradigm](https://mlsanity.com/paper/2605.12271/) (2026)
- [Support-Conditioned Flow Matching Is Kernel Smoothing](https://mlsanity.com/paper/2605.13386/) (2026)
- [Tree-of-Thoughts Reasoning for Text-to-Image In-Context Learning](https://mlsanity.com/paper/2607.07117/) (2026)
- [Transformer-based In-Context Score Operator Learning for Diffusion Models](https://mlsanity.com/paper/2610.13690/) (2026)

## Abstract (arXiv)

We propose Context Diffusion, a diffusion-based framework that enables image generation models to learn from visual examples presented in context. Recent work tackles such in-context learning for image generation, where a query image is provided alongside context examples and text prompts. However, the quality and context fidelity of the generated images deteriorate when the prompt is not present, demonstrating that these models cannot truly learn from the visual context. To address this, we propose a novel framework that separates the encoding of the visual context and the preservation of the desired image layout. This results in the ability to learn from the visual context and prompts, but also from either of them. Furthermore, we enable our model to handle few-shot settings, to effectively address diverse in-context learning scenarios. Our experiments and human evaluation demonstrate that Context Diffusion excels in both in-domain and out-of-domain tasks, resulting in an overall enhancement in image quality and context fidelity compared to counterpart models.