mlsanityData to 14 Oct 2026

Collaborative Diffusion for Multi-Modal Face Generation and Editing

arXiv 2304.10530 ↗First posted 20 Apr 2023cs.CVcited by 116

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion models arise as a powerful generative tool recently. Despite the great progress, existing diffusion models mainly focus on uni-modal control, i.e., the diffusion process is driven by only one modality of condition. To further unleash the users' creativity, it is desirable for the model to be controllable by multiple modalities simultaneously, e.g., generating and editing faces by describing the age (text-driven) while drawing the face shape (mask-driven). In this work, we present Collaborative Diffusion, where pre-trained uni-modal diffusion models collaborate to achieve multi-modal face generation and editing without re-training. Our key insight is that diffusion models driven by different modalities are inherently complementary regarding the latent denoising steps, where bilateral connections can be established upon. Specifically, we propose dynamic diffuser, a meta-network that adaptively hallucinates multi-modal denoising steps by predicting the spatial-temporal influence functions for each pre-trained uni-modal model. Collaborative Diffusion not only collaborates generation capabilities from uni-modal diffusion models, but also integrates multiple uni-modal manipulations to perform multi-modal editing. Extensive qualitative and quantitative experiments demonstrate the superiority of our framework in both image quality and condition consistency.

Key facts. Cited by 116 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 72 of its 76 references that are in the map. First posted on arXiv 20 Apr 2023.

Builds on

72 of 76 references.

Closest ancestors · 12

Same direction, newest first

  1. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  2. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  3. 2022Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model26 Aug
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  5. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  6. 2022Pretraining is All You Need for Image-to-Image Translation25 May
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  10. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance10 Dec
  11. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  12. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  8. 2020Denoising Diffusion Implicit Modelscited by 9,781
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  12. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
38 more
  1. 2016Layer Normalizationcited by 5,669
  2. 2022Generative Adversarial Networkscited by 5,627
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  5. 2013Auto-Encoding Variational Bayescited by 4,053
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  8. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  9. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  10. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  11. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  12. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  13. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  14. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  15. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  16. 2021Alias-Free Generative Adversarial Networkscited by 1,184
  17. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  18. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  19. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  20. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  21. 2016Generative Adversarial Text to Image Synthesiscited by 735
  22. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  23. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 695
  24. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  25. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  26. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 628
  27. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 499
  28. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  29. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  30. 2020InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANscited by 394
  31. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  32. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Modelscited by 280
  33. 2022Diffusion models as plug-and-play priorscited by 231
  34. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  35. 2022Semantic Image Synthesis via Diffusion Modelscited by 132
  36. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  37. 2019Object-driven Text-to-Image Synthesis via Adversarial Trainingcited by 117
  38. 2016Learning What and Where to Drawcited by 105

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 47; 2025: 35; 2026: 20.

116 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  2. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  3. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  4. 2023FreeU: Free Lunch in Diffusion U-Netcited by 196
  5. 2024DF40: Toward Next-Generation Deepfake Detectioncited by 129
  6. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 126
  7. 2023AutoDIR: Automatic All-in-One Image Restoration with Latent Diffusioncited by 100
  8. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 83
  9. 2024Multi-LoRA Composition for Image Generationcited by 80
  10. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 67
  11. 2024Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attentioncited by 59
  12. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 57
38 more
  1. 2024ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preservingcited by 43
  2. 2023DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatarscited by 42
  3. 2024Diffusion Facial Forgery Detectioncited by 41
  4. 2023Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Modelscited by 36
  5. 2025VChain: Chain-of-Visual-Thought for Reasoning in Video Generationcited by 32
  6. 2024It's All About Your Sketch: Democratising Sketch Control in Diffusion Modelscited by 29
  7. 2024in2IN: Leveraging individual Information to Generate Human INteractionscited by 29
  8. 2024Privacy-Preserving Face Recognition Using Trainable Feature Subtractioncited by 24
  9. 2023FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clientscited by 23
  10. 2024Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Functioncited by 19
  11. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Modelscited by 18
  12. 2024Composable Part-Based Manipulationcited by 17
  13. 2024InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generationcited by 16
  14. 202550 Years of Automated Face Recognitioncited by 15
  15. 2025PhysCorr: Dual-Reward DPO for Physics-Constrained Text-to-Video Generation with Automated Preference Selectioncited by 15
  16. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  17. 2024DiffusionAct: Controllable Diffusion Autoencoder for One-shot Face Reenactmentcited by 14
  18. 2024FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-pose, and Facial Expression Featurescited by 14
  19. 202415M Multimodal Facial Image-Text Datasetcited by 14
  20. 2025From Specificity to Generality: Revisiting Generalizable Artifacts in Detecting Face Deepfakescited by 14
  21. 2023Chasing Consistency in Text-to-3D Generation from a Single Imagecited by 13
  22. 2024Towards Localized Fine-Grained Control for Facial Expression Generationcited by 13
  23. 2025MixerMDM: Learnable Composition of Human Motion Diffusion Modelscited by 13
  24. 2025TAMO: Fine-Grained Root Cause Analysis via Tool-Assisted LLM Agent with Multi-Modality Observation Data in Cloud-Native Systemscited by 13
  25. 2024GenFace: A Large-Scale Fine-Grained Face Forgery Benchmark and Cross Appearance-Edge Learningcited by 12
  26. 2024Diffusion-driven GAN Inversion for Multi-Modal Face Image Generationcited by 12
  27. 2024MFCLIP: Multi-modal Fine-grained CLIP for Generalizable Diffusion Face Forgery Detectioncited by 11
  28. 2024Scene Graph Disentanglement and Composition for Generalizable Complex Image Generationcited by 11
  29. 2025RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scalingcited by 10
  30. 2023Controllable 3D Face Generation with Conditional Style Code Diffusioncited by 9
  31. 2024Privacy-Preserving Low-Rank Adaptation against Membership Inference Attacks for Latent Diffusion Modelscited by 9
  32. 2024MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Modelscited by 9
  33. 2024Head360: Learning a Parametric 3D Full-Head for Free-View Synthesis in 360degcited by 9
  34. 2024Face De-identification: State-of-the-art Methods and Comparative Studiescited by 9
  35. 2025Anomaly Detection and Generation with Diffusion Models: A Surveycited by 8
  36. 2024An Efficient and Harmonized Framework for Balanced Cross-Domain Feature Integrationcited by 7
  37. 2024MM2Latent: Text-to-facial image generation and editing in GANs with multimodal assistancecited by 7
  38. 2025LMME3DHF: Benchmarking and Evaluating Multimodal 3D Human Face Generation with LMMscited by 7

50 most cited of 116 All 116 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion Models For Multi-Modal Generative Modeling
  2. 2022Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Models
  3. 2025MoDA: Multi-modal Diffusion Architecture for Talking Head Generation
  4. 2023Cognitively Inspired Cross-Modal Data Generation Using Diffusion Models
  5. 2023Face Aging via Diffusion-based Editing
  6. 2026MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation
  7. 2023Multi-modal Latent Diffusion
  8. 2023DiFace: Cross-Modal Face Recognition through Controlled Diffusion