mlsanityData to 7 Oct 2026

OmniControlNet: Dual-stage Integration for Conditional Image Generation

arXiv 2406.05871 ↗First posted 9 Jun 2024cs.CV cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

We provide a two-way integration for the widely adopted ControlNet by integrating external condition generation algorithms into a single dense prediction method and incorporating its individually trained image generation processes into a single model. Despite its tremendous success, the ControlNet of a two-stage pipeline bears limitations in being not self-contained (e.g. calls the external condition generation algorithms) with a large model redundancy (separately trained models for different types of conditioning inputs). Our proposed OmniControlNet consolidates 1) the condition generation (e.g., HED edges, depth maps, user scribble, and animal pose) by a single multi-tasking dense prediction algorithm under the task embedding guidance and 2) the image generation process for different conditioning types under the textual embedding guidance. OmniControlNet achieves significantly reduced model complexity and redundancy while capable of producing images of comparable quality for conditioned text-to-image generation.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 84 of its 114 references that are in the map. First posted on arXiv 9 Jun 2024.

Builds on

84 of 114 references.

References · 84

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  5. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  13. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  14. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  15. 2021Vision Transformers for Dense Predictioncited by 1,789
  16. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  17. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  18. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  19. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  20. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  21. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  22. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  23. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  24. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  25. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  26. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  27. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  28. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  29. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 913
  30. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  31. 2020Pre-Trained Image Processing Transformercited by 891
  32. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  33. 2016Generative Adversarial Text to Image Synthesiscited by 735
  34. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  35. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 721
  36. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  37. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  38. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 14; 2026: 5.

24 papers.

Sort:

By area:

  1. 2024LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformerscited by 35
  2. 2025DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale Text-to-Image Modelscited by 28
  3. 2024Looped ReLU MLPs May Be All You Need as Practical Programmable Computerscited by 19
  4. 2024HSR-Enhanced Sparse Attention Accelerationcited by 19
  5. 2025X-Dancer: Expressive Music to Human Dance Video Generationcited by 15
  6. 2024Differential Privacy Mechanisms in Neural Tangent Kernel Regressioncited by 14
  7. 2024Provable Differentially Private Computation of the Cross-Attention Mechanismcited by 14
  8. 2025Force Matching with Relativistic Constraints: A Physics-Inspired Approach to Stable and Efficient Generative Modelingcited by 12
  9. 2025Circuit Complexity Bounds for Visual Autoregressive Modelcited by 10
  10. 2025On Computational Limits of FlowAR Models: Expressivity and Efficiencycited by 10
  11. 2025Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Modelscited by 10
  12. 2025Dissecting Submission Limit in Desk-Rejections: A Mathematical Analysis of Fairness in AI Conference Policiescited by 9
12 more
  1. 2025Theoretical Guarantees for High Order Trajectory Refinement in Generative Flowscited by 9
  2. 2025T2VTextBench: A Human Evaluation Benchmark for Textual Control in Video Generation Modelscited by 9
  3. 2025OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlapscited by 9
  4. 2025BideDPO: Conditional Image Generation with Simultaneous Text and Condition Alignmentcited by 9
  5. 2025HOFAR: High-Order Augmentation of Flow Autoregressive Transformerscited by 4
  6. 2026OmniGen-AR: AutoRegressive Any-to-Image Generationcited by 4
  7. 2026DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformerscited by 3
  8. 2026CyCLeGen: Cycle-Consistent Layout Prediction and Image Generation in Vision Foundation Modelscited by 3
  9. 2025CompArt: Operationalizing Aesthetic Alignment in Text-to-Image Generation via Principles of Artcited by 2
  10. 2025Exploring the Equivalence of Closed-Set Generative and Real Data Augmentation in Image Classificationcited by 1
  11. 2026MatchED: Crisp Edge Detection Using End-to-End, Matching-based Supervisioncited by 1
  12. 2026TerraDiT: Point-Conditioned Diffusion Transformer for Satellite Image Synthesiscited by 0