mlsanityData to 7 Oct 2026

X&Fuse: Fusing Visual Information in Text-to-Image Generation

arXiv 2303.01000 ↗First posted 2 Mar 2023cs.CV cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

We introduce X&Fuse, a general approach for conditioning on visual information when generating images from text. We demonstrate the potential of X&Fuse in three different text-to-image generation scenarios. (i) When a bank of images is available, we retrieve and condition on a related image (Retrieve&Fuse), resulting in significant improvements on the MS-COCO benchmark, gaining a state-of-the-art FID score of 6.65 in zero-shot settings. (ii) When cropped-object images are at hand, we utilize them and perform subject-driven generation (Crop&Fuse), outperforming the textual inversion method while being more than x100 faster. (iii) Having oracle access to the image scene (Scene&Fuse), allows us to achieve an FID score of 5.03 on MS-COCO in zero-shot settings. Our experiments indicate that X&Fuse is an effective, easy-to-adapt, simple, and general approach for scenarios in which the model may benefit from additional visual information.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 26 of its 31 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

26 of 31 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
14 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  10. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  11. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  12. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  13. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  14. 2022Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Modelscited by 63

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1; 2025: 2.

6 papers.

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  2. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  3. 20234M: Massively Multimodal Masked Modelingcited by 118
  4. 2023High-fidelity Person-centric Subject-to-Image Synthesiscited by 21
  5. 2025VSC: Visual Search Compositional Text-to-Image Diffusion Modelcited by 7
  6. 2025EQ-TAA: Equivariant Traffic Accident Anticipation via Diffusion-Based Accident Video Synthesiscited by 2