mlsanityData to 13 Oct 2026

When ControlNet Meets Inexplicit Masks: A Case Study of ControlNet on its Contour-following Ability

arXiv 2403.00467 ↗First posted 1 Mar 2024cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

ControlNet excels at creating content that closely matches precise contours in user-provided masks. However, when these masks contain noise, as a frequent occurrence with non-expert users, the output would include unwanted artifacts. This paper first highlights the crucial role of controlling the impact of these inexplicit masks with diverse deterioration levels through in-depth analysis. Subsequently, to enhance controllability with inexplicit masks, an advanced Shape-aware ControlNet consisting of a deterioration estimator and a shape-prior modulation block is devised. The deterioration estimator assesses the deterioration factor of the provided masks. Then this factor is utilized in the modulation block to adaptively modulate the model's contour-following ability, which helps it dismiss the noise part in the inexplicit masks. Extensive experiments prove its effectiveness in encouraging ControlNet to interpret inaccurate spatial conditions robustly rather than blindly following the given contours, suitable for diverse kinds of conditions. We showcase application scenarios like modifying shape priors and composable shape-controllable generation. Codes are available at github.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

39 of 52 references.

References · 39

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,564
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,339
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,000
  6. 2022Classifier-Free Diffusion Guidancecited by 5,785
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,484
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,407
  10. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,490
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,452
  12. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,342
27 more
  1. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  2. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  3. 2016HyperNetworkscited by 1,010
  4. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 777
  5. 2015Holistically-Nested Edge Detectioncited by 592
  6. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  7. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Modelscited by 444
  8. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 359
  9. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 304
  10. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  11. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generationcited by 258
  12. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  13. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 199
  14. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  15. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 97
  16. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89
  17. 2023FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesiscited by 79
  18. 2023LooseControl: Lifting ControlNet for Generalized Depth Conditioningcited by 68
  19. 2023LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generationcited by 65
  20. 2023Controllable Text-to-Image Generation with GPT-4cited by 58
  21. 2023Control3D: Towards Controllable Text-to-3D Generationcited by 46
  22. 2021Towards Light-weight and Real-time Line Segment Detectioncited by 33
  23. 2023SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained Object Insertion and Layout Controlcited by 18
  24. 2023Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generationcited by 12
  25. 2023VisorGPT: Learning Visual Prior via Generative Pre-Trainingcited by 11
  26. 2024Layout-to-Image Generation with Localized Descriptions using ControlNet with Cross-Attention Controlcited by 5
  27. 2023PNT-Edge: Towards Robust Edge Detection with Noisy Labels by Learning Pixel-level Noise Transitionscited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 5.

6 papers.

  1. 2024FlexEdit: Marrying Free-Shape Masks to VLLM for Flexible Image Editingcited by 5
  2. 2025Rethink Sparse Signals for Pose-guided Text-to-image Generationcited by 5
  3. 2025Deeply Dual Supervised learning for melanoma recognitioncited by 0
  4. 2025Edge Detection for Organ Boundaries via Top Down Refinement and SubPixel Upsamplingcited by 0
  5. 2025DualResolution Residual Architecture with Artifact Suppression for Melanocytic Lesion Segmentationcited by 0
  6. 2025VesselRW: Weakly Supervised Subcutaneous Vessel Segmentation via Learned Random Walk Propagationcited by 0