mlsanityData to 7 Oct 2026

ViscoNet: Bridging and Harmonizing Visual and Textual Conditioning for ControlNet

arXiv 2312.03154 ↗First posted 5 Dec 2023cs.CV cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

This paper introduces ViscoNet, a novel one-branch-adapter architecture for concurrent spatial and visual conditioning. Our lightweight model requires trainable parameters and dataset size multiple orders of magnitude smaller than the current state-of-the-art IP-Adapter. However, our method successfully preserves the generative power of the frozen text-to-image (T2I) backbone. Notably, it excels in addressing mode collapse, a pervasive issue previously overlooked. Our novel architecture demonstrates outstanding capabilities in achieving a harmonious visual-text balance, unlocking unparalleled versatility in various human image generation tasks, including pose re-targeting, virtual try-on, stylization, person re-identification, and textile transfer.Demo and code are available from project page https://soon-yau.github.io/visconet/ .

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 62 of its 72 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

62 of 72 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022Generative Adversarial Networkscited by 5,603
38 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  7. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  8. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  9. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  10. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  11. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  12. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  13. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  14. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  15. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  16. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  17. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  18. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  19. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  20. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  21. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  22. 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 411
  23. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 358
  24. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  25. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  26. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  27. 2017Pose Guided Person Image Generationcited by 251
  28. 2023Inpaint Anything: Segment Anything Meets Image Inpaintingcited by 233
  29. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidancecited by 211
  30. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 210
  31. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  32. 2020Semantic Photo Manipulation with a Generative Image Priorcited by 178
  33. 2022Pretraining is All You Need for Image-to-Image Translationcited by 160
  34. 2021EditGAN: High-Precision Semantic Image Editingcited by 156
  35. 2019Progressive Pose Attention Transfer for Person Image Generationcited by 138
  36. 2013Stabilitycited by 119
  37. 2023Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Modelscited by 116
  38. 2022Person Image Synthesis via Denoising Diffusion Modelcited by 104

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 2.

7 papers.

  1. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  2. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
  3. 2024Boosting Camera Motion Control for Video Diffusion Transformerscited by 20
  4. 2024TryOffDiff: Virtual-Try-Off via High-Fidelity Garment Reconstruction using Diffusion Modelscited by 18
  5. 2024TCAN: Animating Human Images with Temporally Consistent Pose Guidance using Diffusion Modelscited by 15
  6. 2025MGT: Extending Virtual Try-Off to Multi-Garment Scenarioscited by 6
  7. 2025SemanticControl: A Training-Free Approach for Handling Loosely Aligned Visual Conditions in ControlNetcited by 0