mlsanityData to 7 Oct 2026

UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild

arXiv 2305.11147 ↗First posted 18 May 2023cs.CV cs.AIcited by 197

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Achieving machine autonomy and human control often represent divergent objectives in the design of interactive AI systems. Visual generative foundation models such as Stable Diffusion show promise in navigating these goals, especially when prompted with arbitrary languages. However, they often fall short in generating images with spatial, structural, or geometric controls. The integration of such controls, which can accommodate various visual conditions in a single unified model, remains an unaddressed challenge. In response, we introduce UniControl, a new generative foundation model that consolidates a wide array of controllable condition-to-image (C2I) tasks within a singular framework, while still allowing for arbitrary language prompts. UniControl enables pixel-level-precise image generation, where visual conditions primarily influence the generated structures and language prompts guide the style and context. To equip UniControl with the capacity to handle diverse visual conditions, we augment pretrained text-to-image diffusion models and introduce a task-aware HyperNet to modulate the diffusion models, enabling the adaptation to different C2I tasks simultaneously. Trained on nine unique C2I tasks, UniControl demonstrates impressive zero-shot generation abilities with unseen visual conditions. Experimental results show that UniControl often surpasses the performance of single-task-controlled methods of comparable model sizes. This control versatility positions UniControl as a significant advancement in the realm of controllable visual generation.

Key facts. Cited by 197 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 61 of its 65 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

61 of 65 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023In-Context Learning Unlocked for Diffusion Models1 May
  2. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  4. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  5. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  6. 2022Pretraining is All You Need for Image-to-Image Translation25 May
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  9. 2022Dual Diffusion Implicit Bridges for Image-to-Image Translation16 Mar
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  11. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  10. 2020Denoising Diffusion Implicit Modelscited by 9,655
  11. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  12. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
38 more
  1. 2023Segment Anythingcited by 8,158
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2022Flow Matching for Generative Modelingcited by 5,426
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  11. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  13. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  14. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  15. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  16. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  17. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  18. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  19. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  20. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  21. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  22. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  23. 2016HyperNetworkscited by 1,003
  24. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  25. 2015Holistically-Nested Edge Detectioncited by 590
  26. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  27. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  28. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  29. 2019Continual learning with hypernetworkscited by 267
  30. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  31. 2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 256
  32. 2022Visual Prompting via Image Inpaintingcited by 248
  33. 2023I2SB: Image-to-Image Schr"odinger Bridgecited by 214
  34. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  35. 2022Let us Build Bridges: Understanding and Extending Diffusion Generative Modelscited by 102
  36. 2021Multi-Task Self-Training for Learning General Representationscited by 85
  37. 2023Efficient Domain Adaptation for Speech Foundation Modelscited by 13
  38. 2023CoBIT: A Contrastive Bi-directional Image-Text Generation Modelcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 22; 2024: 63; 2025: 86; 2026: 26.

197 papers.

Sort:

By area:

  1. 2025VACE: All-in-One Video Creation and Editingcited by 467
  2. 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 411
  3. 2024OmniGen: Unified Image Generationcited by 388
  4. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  5. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 334
  6. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  7. 2023Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylizationcited by 251
  8. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  9. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  10. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196
  11. 2024ControlNeXt: Powerful and Efficient Control for Image and Video Generationcited by 158
  12. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
38 more
  1. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 104
  2. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  3. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 95
  4. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  5. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  6. 2025DreamO: A Unified Framework for Image Customizationcited by 82
  7. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  8. 2023ImagenHub: Standardizing the evaluation of conditional image generation modelscited by 75
  9. 2025Ovis-U1 Technical Reportcited by 65
  10. 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Modelcited by 64
  11. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  12. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  13. 2024ACE: All-round Creator and Editor Following Instructions via Diffusion Transformercited by 56
  14. 2025OminiControl2: Efficient Conditioning for Diffusion Transformerscited by 56
  15. 2024λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Spacecited by 51
  16. 2023Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Modelscited by 48
  17. 2024ControlAR: Controllable Image Generation with Autoregressive Modelscited by 48
  18. 2024Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Taskscited by 47
  19. 2024One Diffusion to Generate Them Allcited by 41
  20. 2024FonTS: Text Rendering with Typography and Style Controlscited by 40
  21. 2025EditAR: Unified Conditional Generation with Autoregressive Modelscited by 40
  22. 2023SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editingcited by 37
  23. 2025DICEPTION: A Generalist Diffusion Model for Visual Perceptual Taskscited by 37
  24. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  25. 2025Efficient Diffusion Models: A Surveycited by 35
  26. 2024SimGen: Simulator-conditioned Driving Scene Generationcited by 34
  27. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  28. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  29. 2024DreamOmni: Unified Image Generation and Editingcited by 29
  30. 2023ControlNet-XS: Rethinking the Control of Text-to-Image Diffusion Models as Feedback-Control Systemscited by 28
  31. 2024GlyphDraw2: Automatic Generation of Complex Glyph Posters with Diffusion Models and Large Language Modelscited by 28
  32. 2025UniCombine: Unified Multi-Conditional Combination with Diffusion Transformercited by 25
  33. 2024SmartControl: Enhancing ControlNet for Handling Rough Visual Conditionscited by 24
  34. 2024OmniControlNet: Dual-stage Integration for Conditional Image Generationcited by 24
  35. 2025X2Edit: Revisiting Arbitrary-Instruction Image Editing through Self-Constructed Data and Task-Aware Representation Learningcited by 24
  36. 2023A Survey of Diffusion Based Image Generation Models: Issues and Their Solutionscited by 23
  37. 2024Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Modelscited by 23
  38. 2023Image Super-Resolution with Text Prompt Diffusioncited by 22

50 most cited of 197 All 197 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling
  2. 2026Large Language Models are Universal Reasoners for Visual Generation
  3. 2025VisualCloze: A Universal Image Generation Framework via Visual In-Context Learning
  4. 2024DiffusionAgent: Navigating Expert Models for Agentic Image Generation
  5. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation
  6. 2026Wan-Image: Pushing the Boundaries of Generative Visual Intelligence
  7. 2025UniFusion: Vision-Language Model as Unified Encoder in Image Generation
  8. 2026UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation