# Improving Diffusion-Based Image Synthesis with Context Prediction

arXiv: https://arxiv.org/abs/2401.02015 · HTML: https://mlsanity.com/paper/2401.02015/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 91 of its 111 references that are in the map.
- First posted on arXiv 4 Jan 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 91)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)

## Built on by (most cited first, 30 of 32)

- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [MMaDA: Multimodal Large Diffusion Language Models](https://mlsanity.com/paper/2505.15809/) (2025)
- [Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs](https://mlsanity.com/paper/2401.11708/) (2024)
- [GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing](https://mlsanity.com/paper/2407.05600/) (2024)
- [DiT4Edit: Diffusion Transformer for Image Editing](https://mlsanity.com/paper/2411.03286/) (2024)
- [EditWorld: Simulating World Dynamics for Instruction-Following Image Editing](https://mlsanity.com/paper/2405.14785/) (2024)
- [R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation](https://mlsanity.com/paper/2505.23493/) (2025)
- [Image Generation Diversity Issues and How to Tame Them](https://mlsanity.com/paper/2411.16171/) (2024)
- [Distribution-Aware Data Expansion with Diffusion Models](https://mlsanity.com/paper/2403.06741/) (2024)
- [RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models](https://mlsanity.com/paper/2402.12908/) (2024)
- [Binding-Adaptive Diffusion Models for Structure-Based Drug Design](https://mlsanity.com/paper/2402.18583/) (2024)
- [Structure-Guided Adversarial Training of Diffusion Models](https://mlsanity.com/paper/2402.17563/) (2024)
- [Federated Learning for Diffusion Models](https://mlsanity.com/paper/2503.06426/) (2025)
- [REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models](https://mlsanity.com/paper/2408.02231/) (2024)
- [Draw Like an Artist: Complex Scene Generation with Diffusion Model via Composition, Painting, and Retouching](https://mlsanity.com/paper/2408.13858/) (2024)
- [Preacher: Paper-to-Video Agentic System](https://mlsanity.com/paper/2508.09632/) (2025)
- [HouseTune: Two-Stage Floorplan Generation with LLM Assistance](https://mlsanity.com/paper/2411.12279/) (2024)
- [Towards Improved Text-Aligned Codebook Learning: Multi-Hierarchical Codebook-Text Alignment with Long Text](https://mlsanity.com/paper/2503.01261/) (2025)
- [HyperDet: Generalizable Detection of Synthesized Images by Generating and Merging A Mixture of Hyper LoRAs](https://mlsanity.com/paper/2410.06044/) (2024)
- [THEval. Evaluation Framework for Talking Head Video Generation](https://mlsanity.com/paper/2511.04520/) (2025)
- [PatchScaler: An Efficient Patch-Independent Diffusion Model for Image Super-Resolution](https://mlsanity.com/paper/2405.17158/) (2024)
- [Tex-ViT: A Generalizable, Robust, Texture-based dual-branch cross-attention deepfake detector](https://mlsanity.com/paper/2408.16892/) (2024)
- [Learning Straight Flows by Learning Curved Interpolants](https://mlsanity.com/paper/2503.20719/) (2025)
- [Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learning](https://mlsanity.com/paper/2505.01822/) (2025)
- [A Difference-in-Difference Approach to Detecting AI-Generated Images](https://mlsanity.com/paper/2602.23732/) (2026)
- [Contextualized Diffusion Models for Text-Guided Image and Video Generation](https://mlsanity.com/paper/2402.16627/) (2024)
- [Agentic Flow Steering and Parallel Rollout Search for Spatially Grounded Text-to-Image Generation](https://mlsanity.com/paper/2603.18627/) (2026)
- [RIHA: Report-Image Hierarchical Alignment for Radiology Report Generation](https://mlsanity.com/paper/2604.27559/) (2026)
- [SciIR: A Large-scale Training Dataset and Benchmark for Scientific Image Reasoning Generation](https://mlsanity.com/paper/2606.30124/) (2026)
- [In the Blink of an Eye: Instant Game Map Editing using a Generative-AI Smart Brush](https://mlsanity.com/paper/2503.19793/) (2025)

## Abstract (arXiv)

Diffusion models are a new class of generative models, and have dramatically promoted image generation with unprecedented quality and diversity. Existing diffusion models mainly try to reconstruct input image from a corrupted one with a pixel-wise or feature-wise constraint along spatial axes. However, such point-based reconstruction may fail to make each predicted pixel/feature fully preserve its neighborhood context, impairing diffusion-based image synthesis. As a powerful source of automatic supervisory signal, context has been well studied for learning representations. Inspired by this, we for the first time propose ConPreDiff to improve diffusion-based image synthesis with context prediction. We explicitly reinforce each point to predict its neighborhood context (i.e., multi-stride features/tokens/pixels) with a context decoder at the end of diffusion denoising blocks in training stage, and remove the decoder for inference. In this way, each point can better reconstruct itself by preserving its semantic connections with neighborhood context. This new paradigm of ConPreDiff can generalize to arbitrary discrete and continuous diffusion backbones without introducing extra parameters in sampling procedure. Extensive experiments are conducted on unconditional image generation, text-to-image generation and image inpainting tasks. Our ConPreDiff consistently outperforms previous methods and achieves a new SOTA text-to-image generation results on MS-COCO, with a zero-shot FID score of 6.21.