mlsanityData to 7 Oct 2026

Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models

arXiv 2207.13038 ↗First posted 26 Jul 2022cs.CVcited by 63

On the map: Retrieval-augmented generation → Multimodal RAG

also in 1 more
Abstract (arXiv)

Novel architectures have recently improved generative image synthesis leading to excellent visual quality in various tasks. Of particular note is the field of ``AI-Art'', which has seen unprecedented growth with the emergence of powerful multimodal models such as CLIP. By combining speech and image synthesis models, so-called ``prompt-engineering'' has become established, in which carefully selected and composed sentences are used to achieve a certain visual style in the synthesized image. In this note, we present an alternative approach based on retrieval-augmented diffusion models (RDMs). In RDMs, a set of nearest neighbors is retrieved from an external database during training for each training instance, and the diffusion model is conditioned on these informative samples. During inference (sampling), we replace the retrieval database with a more specialized database that contains, for example, only images of a particular visual style. This provides a novel way to prompt a general trained model after training and thereby specify a particular visual style. As shown by our experiments, this approach is superior to specifying the visual style within the text prompt. We open-source code and model weights at https://github.com/CompVis/latent-diffusion .

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 25 of its 30 references that are in the map. First posted on arXiv 26 Jul 2022.

Builds on

25 of 30 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Retrieval Augmented Classification for Long-Tail Visual Recognition22 Feb

Other references · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
12 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  3. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  4. 2020Nearest Neighbor Machine Translationcited by 209
  5. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 181
  6. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  7. 2021Instance-Conditioned GANcited by 119
  8. 2022The ArtBench Dataset: Benchmarking Generative Models with Artworkscited by 64
  9. 2020RetrieveGAN: Image Synthesis via Differentiable Patch Retrievalcited by 35
  10. 2021RetrievalFuse: Neural 3D Scene Reconstruction with a Databasecited by 28
  11. 2021GNN-LM: Language Modeling based on Global Contexts via GNNcited by 27
  12. 2020Texture Memory-Augmented Deep Patch-Based Image Inpaintingcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 19; 2024: 25; 2025: 11; 2026: 5.

63 papers.

Sort:

By area:

  1. 2022Diffusion Models in Vision: A Surveycited by 1,008
  2. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  3. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  4. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  5. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  6. 2023AGIQA-3K: An Open Database for AI-Generated Image Quality Assessmentcited by 142
  7. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animationcited by 136
  8. 2024Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Comparecited by 66
  9. 2022Efficient Diffusion Models for Vision: A Surveycited by 50
  10. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  11. 2025Text2Earth: Unlocking Text-driven Remote Sensing Image Generation with a Global-Scale Dataset and a Foundation Modelcited by 46
  12. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 42
38 more
  1. 2024Curriculum Direct Preference Optimization for Diffusion and Consistency Modelscited by 36
  2. 2024AIGIQA-20K: A Large Database for AI-Generated Image Quality Assessmentcited by 33
  3. 2023IMPRESS: Evaluating the Resilience of Imperceptible Perturbations Against Unauthorized Data Usage in Diffusion-Based Generative AIcited by 32
  4. 2024MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Modelcited by 29
  5. 2024SeD: Semantic-Aware Discriminator for Image Super-Resolutioncited by 26
  6. 2024Large Multi-modality Model Assisted AI-Generated Image Quality Assessmentcited by 24
  7. 2023DifFSS: Diffusion Model for Few-Shot Semantic Segmentationcited by 22
  8. 2024CPR: Retrieval Augmented Generation for Copyright Protectioncited by 22
  9. 2024Q-Refine: A Perceptual Quality Refiner for AI-Generated Imagecited by 21
  10. 2024NTIRE 2024 Quality Assessment of AI-Generated Content Challengecited by 20
  11. 2024A Survey on Diffusion Models for Recommender Systemscited by 20
  12. 2025RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learningcited by 20
  13. 2024Shake to Leak: Fine-tuning Diffusion Models Can Amplify the Generative Privacy Riskcited by 17
  14. 2023Diffusion Models in NLP: A Surveycited by 16
  15. 2023Enhancing Diffusion Models with 3D Perspective Geometry Constraintscited by 16
  16. 2023MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generationcited by 14
  17. 2024Retrieval-Augmented Score Distillation for Text-to-3D Generationcited by 14
  18. 2023Null-text Guidance in Diffusion Models is Secretly a Cartoon-style Creatorcited by 12
  19. 2025Image Quality Assessment: From Human to Machine Preferencecited by 12
  20. 2023PSCR: Patches Sampling-based Contrastive Regression for AIGC Image Quality Assessmentcited by 10
  21. 2024MuseumMaker: Continual Style Customization without Catastrophic Forgettingcited by 10
  22. 2023Prompt Evolution for Generative AI: A Classifier-Guided Approachcited by 9
  23. 2024Vision-Language Consistency Guided Multi-modal Prompt Learning for Blind AI Generated Image Quality Assessmentcited by 9
  24. 2024RAGDiffusion: Faithful Cloth Generation via External Knowledge Assimilationcited by 9
  25. 2023Vision + Language Applications: A Surveycited by 8
  26. 2024TIER: Text-Image Encoder-based Regression for AIGC Image Quality Assessmentcited by 7
  27. 2023X&Fuse: Fusing Visual Information in Text-to-Image Generationcited by 6
  28. 2024Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineeringcited by 6
  29. 2024CMC-Bench: Towards a New Paradigm of Visual Signal Compressioncited by 6
  30. 2023PARASOL: Parametric Style Control for Diffusion Image Synthesiscited by 5
  31. 2025Using GUI Agent for Electronic Design Automationcited by 5
  32. 2024Fine-Tuning Text-To-Image Diffusion Models for Class-Wise Spurious Feature Generationcited by 4
  33. 2025Q-Fusion: Diffusing Quantum Circuitscited by 4
  34. 2023The Cultivated Practices of Text-to-Image Generationcited by 3
  35. 2023On quantifying and improving realism of images generated with diffusioncited by 3
  36. 2024Leapfrog Latent Consistency Model (LLCM) for Medical Images Generationcited by 3
  37. 2023VoiceExtender: Short-utterance Text-independent Speaker Verification with Guided Diffusion Modelcited by 2
  38. 2024ExactDreamer: High-Fidelity Text-to-3D Content Creation via Exact Score Matchingcited by 2

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Best Prompts for Text-to-Image Models and How to Find Them
  2. 2024Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models
  3. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models
  4. 2024Visual Style Prompting with Swapping Self-Attention
  5. 2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
  6. 2025The Cow of Rembrandt - Analyzing Artistic Prompt Interpretation in Text-to-Image Models
  7. 2023Text-driven Visual Synthesis with Latent Diffusion Prior
  8. 2022Language Does More Than Describe: On The Lack Of Figurative Speech in Text-To-Image Models