Data to 7 Oct 2026

ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback

arXiv 2404.07987 ↗First posted 11 Apr 2024cs.CV cs.AI cs.LGcited by 153

On the map: Diffusion models → Image generation, text-to-image and editing

also in 1 more

LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

To enhance the controllability of text-to-image diffusion models, existing efforts like ControlNet incorporated image-based conditional controls. In this paper, we reveal that existing methods still face significant challenges in generating images that align with the image conditional controls. To this end, we propose ControlNet++, a novel approach that improves controllable generation by explicitly optimizing pixel-level cycle consistency between generated images and conditional controls. Specifically, for an input conditional control, we use a pre-trained discriminative reward model to extract the corresponding condition of the generated images, and then optimize the consistency loss between the input conditional control and extracted condition. A straightforward implementation would be generating images from random noises and then calculating the consistency loss, but such an approach requires storing gradients for multiple sampling timesteps, leading to considerable time and memory costs. To address this, we introduce an efficient reward strategy that deliberately disturbs the input images by adding noise, and then uses the single-step denoised images for reward fine-tuning. This avoids the extensive costs associated with image sampling, allowing for more efficient reward fine-tuning. Extensive experiments show that ControlNet++ significantly improves controllability under various conditional controls. For example, it achieves improvements over ControlNet by 11.1% mIoU, 13.4% SSIM, and 7.6% RMSE, respectively, for segmentation mask, line-art edge, and depth conditions. All the code, models, demo and organized data have been open sourced on our Github Repo.

Key facts. Cited by 153 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 63 of its 69 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

63 of 69 references.

Closest ancestors · 16

Same direction, newest first

4 more

Other references · 47

2015Deep Residual Learning for Image Recognitioncited by 51,809
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2020Denoising Diffusion Probabilistic Modelscited by 20,851
2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
2022Training language models to follow instructions with human feedbackcited by 15,712
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2020Denoising Diffusion Implicit Modelscited by 9,655
2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185

35 more

2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
2022Classifier-Free Diffusion Guidancecited by 5,717
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
2021Zero-Shot Text-to-Image Generationcited by 4,276
2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
2021Variational Diffusion Modelscited by 1,145
2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
2023Training Diffusion Models with Reinforcement Learningcited by 939
2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
2016Training Deep Nets with Sublinear Memory Costcited by 825
2015Holistically-Nested Edge Detectioncited by 590
2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Modelscited by 441
2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
2019Kornia: an Open Source Differentiable Computer Vision Library for PyTorchcited by 231
2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 123
2023HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generationcited by 104

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 35; 2025: 80; 2026: 38.

153 papers.

Sort:

By area:

2024OmniGen: Unified Image Generationcited by 388
2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
2025Seedream 4.0: Toward Next-generation Multimodal Image Generationcited by 233
2025TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Modelscited by 204
2024ControlNeXt: Powerful and Efficient Control for Image and Video Generationcited by 158
2024Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesiscited by 153
2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
2025EasyControl: Adding Efficient and Flexible Control for Diffusion Transformercited by 118
2025SkyReels-A2: Compose Anything in Video Diffusion Transformerscited by 72
2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Modelcited by 64
2025RewardDance: Reward Scaling in Visual Generationcited by 63
2025Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Modelcited by 59

38 more

2025OminiControl2: Efficient Conditioning for Diffusion Transformerscited by 56
2024ControlAR: Controllable Image Generation with Autoregressive Modelscited by 48
2024Diffusion Self-Distillation for Zero-Shot Customized Image Generationcited by 46
2025ControlVLA: Few-shot Object-centric Adaptation for Pre-trained Vision-Language-Action Modelscited by 43
2025EditAR: Unified Conditional Generation with Autoregressive Modelscited by 40
2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
20243DIS: Depth-Driven Decoupled Instance Synthesis for Text-to-Image Generationcited by 36
2025Efficient Diffusion Models: A Surveycited by 35
2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
2024DreamOmni: Unified Image Generation and Editingcited by 29
2025DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale Text-to-Image Modelscited by 28
2025Omni-Effects: Unified and Spatially-Controllable Visual Effects Generationcited by 27
2025EasyText: Controllable Diffusion Transformer for Multilingual Text Renderingcited by 25
2025OneReward: Unified Mask-Guided Image Generation via Multi-Task Human Preference Learningcited by 22
2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
2026BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Maskscited by 19
2024IPAdapter-Instruct: Resolving Ambiguity in Image-based Conditioning using Instruct Promptscited by 18
2025SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editingcited by 18
2025SCALAR: Scale-wise Controllable Visual Autoregressive Learningcited by 18
2024TopoCellGen: Generating Histopathology Cell Topology with a Diffusion Modelcited by 16
2025WordCon: Word-level Typography Control in Scene Text Renderingcited by 16
2025Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Renderingcited by 16
2024Ctrl-U: Robust Conditional Image Generation via Uncertainty-aware Reward Modelingcited by 14
2025Ingredients: Blending Custom Photos with Video Diffusion Transformerscited by 13
2025Mod-Adapter: Tuning-Free and Versatile Multi-concept Personalization via Modulation Adaptercited by 13
2024Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generationcited by 12
2025On the Limitations of Vision-Language Models in Understanding Image Transformscited by 12
2025TextFlux: An OCR-Free DiT Model for High-Fidelity Multilingual Scene Text Synthesiscited by 12
2024Video Diffusion Transformers are In-Context Learnerscited by 11
2025Panorama Generation From NFoV Image Done Rightcited by 11
2025DC-ControlNet: Decoupling Inter- and Intra-Element Conditions in Image Generation with Diffusion Modelscited by 10
2025Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Controlcited by 10
2024UNIC-Adapter: Unified Image-instruction Adapter with Multi-modal Transformer for Image Generationcited by 9
2025InterLCM: Low-Quality Images as Intermediate States of Latent Consistency Models for Effective Blind Face Restorationcited by 9
2025PRISM: A Unified Framework for Photorealistic Reconstruction and Intrinsic Scene Modelingcited by 9
2025MultiRef: Controllable Image Generation with Multiple Visual Referencescited by 9
2025BideDPO: Conditional Image Generation with Simultaneous Text and Condition Alignmentcited by 9
2024SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Modelscited by 8

50 most cited of 153 All 153 (JSON)

Similar, not cited

Close in wording, no citation either way