mlsanityData to 7 Oct 2026

ReCo: Region-Controlled Text-to-Image Generation

arXiv 2211.15518 ↗First posted 23 Nov 2022cs.CVcited by 177

Not on the map. Why?

Cited by 89 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, large-scale text-to-image (T2I) models have shown impressive performance in generating high-fidelity images, but with limited controllability, e.g., precisely specifying the content in a specific region with a free-form text description. In this paper, we propose an effective technique for such regional control in T2I generation. We augment T2I models' inputs with an extra set of position tokens, which represent the quantized spatial coordinates. Each region is specified by four position tokens to represent the top-left and bottom-right corners, followed by an open-ended natural language regional description. Then, we fine-tune a pre-trained T2I model with such new input interface. Our model, dubbed as ReCo (Region-Controlled T2I), enables the region control for arbitrary objects described by open-ended regional texts rather than by object labels from a constrained category set. Empirically, ReCo achieves better image quality than the T2I model strengthened by positional words (FID: 8.82->7.36, SceneFID: 15.54->6.51 on COCO), together with objects being more accurately placed, amounting to a 20.40% region classification accuracy improvement on COCO. Furthermore, we demonstrate that ReCo can better control the object count, spatial relationship, and region attributes such as color/size, with the free-form regional description. Human evaluation on PaintSkill shows that ReCo is +19.28% and +17.21% more accurate in generating images with correct object count and spatial relationship than the T2I model.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). Builds on 45 of its 53 references that are in the map. First posted on arXiv 23 Nov 2022.

Builds on

45 of 53 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
33 more
  1. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  2. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  3. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  4. 2016Generative Adversarial Text to Image Synthesiscited by 735
  5. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  6. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  7. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  8. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  9. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  10. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  11. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  12. 2018Image Generation from Scene Graphscited by 324
  13. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  14. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  15. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  16. 2019Connecting Vision and Language with Localized Narrativescited by 225
  17. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  18. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  19. 2019Object-driven Text-to-Image Synthesis via Adversarial Trainingcited by 117
  20. 2018Image Generation from Layoutcited by 105
  21. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  22. 2019Image Synthesis From Reconfigurable Layout and Stylecited by 85
  23. 2021Multimodal Conditional Image Synthesis with Product-of-Experts GANscited by 75
  24. 2022Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesiscited by 75
  25. 2020Object-Centric Image Generation from Layoutscited by 57
  26. 2022Modeling Image Composition for Complex Scene Generationcited by 43
  27. 2019Generating Multiple Objects at Spatially Distinct Locationscited by 41
  28. 2020Text-to-Image Generation Grounded by Fine-Grained User Attentioncited by 36
  29. 2020BachGAN: High-Resolution Image Synthesis from Salient Object Layoutcited by 22
  30. 2019Controlling Style and Semantics in Weakly-Supervised Image Generationcited by 11
  31. 2021AttrLostGAN: Attribute Controlled Image Synthesis from Reconfigurable Layout and Stylecited by 11
  32. 2020Image-to-Image Translation with Text Guidancecited by 10
  33. 2022DT2I: Dense Text-to-Image Generation from Region Descriptionscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 36; 2024: 78; 2025: 52; 2026: 11.

177 papers.

Sort:

By area:

  1. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  2. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  3. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  4. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 280
  5. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  6. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  7. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
  8. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
  9. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 138
  10. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  11. 20234M: Massively Multimodal Masked Modelingcited by 118
  12. 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
38 more
  1. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  2. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  3. 2023A-STAR: Test-time Attention Segregation and Retention for Text-to-image Synthesiscited by 78
  4. 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generationcited by 68
  5. 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Modelcited by 64
  6. 2023Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Modelscited by 58
  7. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  8. 2023GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generationcited by 56
  9. 2023ControlRoom3D: Room Generation using Semantic Proxy Roomscited by 54
  10. 2024Boosting Consistency in Story Visualization with Rich-Contextual Conditional Diffusion Modelscited by 54
  11. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Promptscited by 53
  12. 2023Photoswap: Personalized Subject Swapping in Imagescited by 47
  13. 2024Make It Count: Text-to-Image Generation with an Accurate Number of Objectscited by 47
  14. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 44
  15. 2024SDDGR: Stable Diffusion-based Deep Generative Replay for Class Incremental Object Detectioncited by 42
  16. 2024HanDiffuser: Text-to-Image Generation With Realistic Hand Appearancescited by 42
  17. 2024Diffusion Facial Forgery Detectioncited by 41
  18. 2024SceneCraft: Layout-Guided 3D Scene Generationcited by 40
  19. 2023Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generationcited by 38
  20. 2024Compositional Text-to-Image Generation with Dense Blob Representationscited by 38
  21. 2024Region-Aware Text-to-Image Generation via Hard Binding and Soft Refinementcited by 37
  22. 2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37
  23. 20243DIS: Depth-Driven Decoupled Instance Synthesis for Text-to-Image Generationcited by 36
  24. 2024MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesiscited by 35
  25. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  26. 2024PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generationcited by 34
  27. 2024Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidancecited by 33
  28. 2024DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perceptioncited by 32
  29. 2024IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generationcited by 32
  30. 2024AeroGen: Enhancing Remote Sensing Object Detection with Diffusion-Driven Data Generationcited by 32
  31. 2023Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedbackcited by 31
  32. 2025Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuningcited by 31
  33. 2024Getting it Right: Improving Spatial Consistency in Text-to-Image Modelscited by 30
  34. 2024Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Modelscited by 30
  35. 2024Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidancecited by 30
  36. 2023SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generationcited by 29
  37. 2023BiomedJourney: Counterfactual Biomedical Image Generation by Instruction-Learning from Multimodal Patient Journeyscited by 29
  38. 2025DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale Text-to-Image Modelscited by 28

50 most cited of 177 All 177 (JSON)