mlsanityData to 7 Oct 2026

Re-Imagen: Retrieval-Augmented Text-to-Image Generator

arXiv 2209.14491 ↗First posted 29 Sep 2022cs.CV cs.AI cs.LGcited by 208

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Research on text-to-image generation has witnessed significant progress in generating diverse and photo-realistic images, driven by diffusion and auto-regressive models trained on large-scale image-text data. Though state-of-the-art models can generate high-quality images of common entities, they often have difficulty generating images of uncommon entities, such as `Chortai (dog)' or `Picarones (food)'. To tackle this issue, we present the Retrieval-Augmented Text-to-Image Generator (Re-Imagen), a generative model that uses retrieved information to produce high-fidelity and faithful images, even for rare or unseen entities. Given a text prompt, Re-Imagen accesses an external multi-modal knowledge base to retrieve relevant (image, text) pairs and uses them as references to generate the image. With this retrieval step, Re-Imagen is augmented with the knowledge of high-level semantics and low-level visual details of the mentioned entities, and thus improves its accuracy in generating the entities' visual appearances. We train Re-Imagen on a constructed dataset containing (image, text, retrieval) triples to teach the model to ground on both text prompt and retrieval. Furthermore, we develop a new sampling strategy to interleave the classifier-free guidance for text and retrieval conditions to balance the text and retrieval alignment. Re-Imagen achieves significant gain on FID score over COCO and WikiImage. To further evaluate the capabilities of the model, we introduce EntityDrawBench, a new benchmark that evaluates image generation for diverse entities, from frequent to rare, across multiple object categories including dogs, foods, landmarks, birds, and characters. Human evaluation on EntityDrawBench shows that Re-Imagen can significantly improve the fidelity of generated images, especially on less frequent entities.

Key facts. Cited by 208 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 40 of its 52 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

40 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Retrieval Augmented Classification for Long-Tail Visual Recognition22 Feb

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
27 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  6. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  11. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  12. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  13. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  14. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  15. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  16. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  17. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  18. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  19. 2016Generative Visual Manipulation on the Natural Image Manifoldcited by 348
  20. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  21. 2021HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editingcited by 208
  22. 2021High-Fidelity GAN Inversion for Image Attribute Editingcited by 197
  23. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  24. 2021WebQA: Multihop and Multimodal QAcited by 109
  25. 2021RetrievalFuse: Neural 3D Scene Reconstruction with a Databasecited by 28
  26. 2020Texture Memory-Augmented Deep Patch-Based Image Inpaintingcited by 22
  27. 2022Memory-Driven Text-to-Image Generationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 48; 2024: 73; 2025: 56; 2026: 22.

208 papers.

Sort:

By area:

  1. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  2. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  3. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  4. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  5. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  6. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  7. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  8. 2024OmniGen: Unified Image Generationcited by 388
  9. 2023DIRE for Diffusion-Generated Image Detectioncited by 386
  10. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  11. 2023Tool Learning with Foundation Modelscited by 302
  12. 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Modelcited by 282
38 more
  1. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  2. 2023Generating Images with Multimodal Language Modelscited by 259
  3. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  4. 2025Transfer between Modalities with MetaQueriescited by 231
  5. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  6. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 210
  7. 2023Glaze: Protecting Artists from Style Mimicry by Text-to-Image Modelscited by 200
  8. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  9. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  10. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  11. 2025Less-to-More Generalization: Unlocking More Controllability by In-Context Generationcited by 153
  12. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  13. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 123
  14. 2023Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Modelscited by 116
  15. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  16. 2024UniReal: Universal Image Generation and Editing via Learning Real-world Dynamicscited by 100
  17. 2023Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoningcited by 91
  18. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  19. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  20. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  21. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  22. 2022Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Modelscited by 69
  23. 2024Instruct-Imagen: Image Generation with Multi-modal Instructioncited by 65
  24. 2023GenerateCT: Text-Conditional Generation of 3D Chest CT Volumescited by 64
  25. 2023Inserting Anybody in Diffusion Models via Celeb Basiscited by 63
  26. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generationcited by 63
  27. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  28. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  29. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 57
  30. 2024λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Spacecited by 51
  31. 2022Efficient Diffusion Models for Vision: A Surveycited by 50
  32. 2023DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Imagecited by 48
  33. 2023Photoswap: Personalized Subject Swapping in Imagescited by 47
  34. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  35. 2023RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarioscited by 46
  36. 2023Detecting Images Generated by Deep Diffusion Models using their Local Intrinsic Dimensionalitycited by 44
  37. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  38. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42

50 most cited of 208 All 208 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025TextTIGER: Text-based Intelligent Generation with Entity Prompt Refinement for Text-to-Image Generation
  2. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
  3. 2025Category-level Text-to-Image Retrieval Improved: Bridging the Domain Gap with Diffusion Models and Vision Encoders
  4. 2022Multi-Concept Customization of Text-to-Image Diffusion
  5. 2025POET: Preference Optimization for Enhanced Text-to-Image Generation
  6. 2024PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction
  7. 2024What If We Recaption Billions of Web Images with LLaMA-3?
  8. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation