mlsanityData to 7 Oct 2026

GLIGEN: Open-Set Grounded Text-to-Image Generation

arXiv 2301.07093 ↗First posted 17 Jan 2023cs.CV cs.AI cs.CLcited by 774

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Large-scale text-to-image diffusion models have made amazing advances. However, the status quo is to use text input alone, which can impede controllability. In this work, we propose GLIGEN, Grounded-Language-to-Image Generation, a novel approach that builds upon and extends the functionality of existing pre-trained text-to-image diffusion models by enabling them to also be conditioned on grounding inputs. To preserve the vast concept knowledge of the pre-trained model, we freeze all of its weights and inject the grounding information into new trainable layers via a gated mechanism. Our model achieves open-world grounded text2img generation with caption and bounding box condition inputs, and the grounding ability generalizes well to novel spatial configurations and concepts. GLIGEN's zero-shot performance on COCO and LVIS outperforms that of existing supervised layout-to-image baselines by a large margin.

Key facts. Cited by 774 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 66 of its 80 references that are in the map. First posted on arXiv 17 Jan 2023.

Builds on

66 of 80 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Shifted Diffusion for Text-to-image Generation24 Nov
  2. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  3. 2022Lafite2: Few-shot Text-to-Image Generation25 Oct
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  7. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2017Mask R-CNNcited by 7,259
38 more
  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  6. 2022A ConvNet for the 2020scited by 3,545
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  13. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  14. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  15. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  16. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  17. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  18. 2021Grounded Language-Image Pre-trainingcited by 1,147
  19. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  20. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  21. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 691
  22. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  23. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  24. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  25. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  26. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  27. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  28. 2018Image Generation from Scene Graphscited by 324
  29. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  30. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  31. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  32. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
  33. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  34. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  35. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  36. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  37. 2019Object-driven Text-to-Image Synthesis via Adversarial Trainingcited by 117
  38. 2018Image Generation from Layoutcited by 105

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 164; 2024: 270; 2025: 216; 2026: 124.

774 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  4. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  5. 2023Segment Everything Everywhere All at Oncecited by 546
  6. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  7. 2023Towards Universal Fake Image Detectors that Generalize Across Generative Modelscited by 483
  8. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  9. 2024OmniGen: Unified Image Generationcited by 388
  10. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 358
  11. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  12. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
38 more
  1. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 324
  2. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  3. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  4. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 280
  5. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  6. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  7. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  8. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  9. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  10. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  11. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  12. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 197
  13. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  14. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  15. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  16. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 182
  17. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  18. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
  19. 2023StableVITON: Learning Semantic Correspondence with Latent Diffusion Model for Virtual Try-Oncited by 173
  20. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  21. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  22. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 163
  23. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 163
  24. 2023Dense Text-to-Image Generation with Attention Modulationcited by 162
  25. 2024ControlNeXt: Powerful and Efficient Control for Image and Video Generationcited by 158
  26. 2023StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Modelscited by 154
  27. 2023Manifold Preserving Guided Diffusioncited by 153
  28. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
  29. 2023Cones: Concept Neurons in Diffusion Models for Customized Generationcited by 150
  30. 2023Localizing Object-level Shape Variations with Text-to-Image Diffusion Modelscited by 142
  31. 2024MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidancecited by 142
  32. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  33. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  34. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 138
  35. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 136
  36. 2023Cross-Image Attention for Zero-Shot Appearance Transfercited by 135
  37. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 132
  38. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129

50 most cited of 774 All 774 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
  2. 2024UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion
  3. 2026LGTM: Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulation
  4. 2023GlyphDiffusion: Text Generation as Image Generation
  5. 2025Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis
  6. 2023VGDiffZero: Text-to-image Diffusion Models Can Be Zero-shot Visual Grounders
  7. 2024Refining Text-to-Image Generation: Towards Accurate Training-Free Glyph-Enhanced Image Generation
  8. 2023GlyphDraw: Seamlessly Rendering Text with Intricate Spatial Structures in Text-to-Image Generation