mlsanityData to 14 Oct 2026

Mixture of Global and Local Experts with Diffusion Transformer for Controllable Face Generation

arXiv 2509.00428 ↗First posted 30 Aug 2025cs.CVcited by 4

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Controllable face generation poses critical challenges in generative modeling due to the intricate balance required between semantic controllability and photorealism. While existing approaches struggle with disentangling semantic controls from generation pipelines, we revisit the architectural potential of Diffusion Transformers (DiTs) through the lens of expert specialization. This paper introduces Face-MoGLE, a novel framework featuring: (1) Semantic-decoupled latent modeling through mask-conditioned space factorization, enabling precise attribute manipulation; (2) A mixture of global and local experts that captures holistic structure and region-level semantics for fine-grained controllability; (3) A dynamic gating network producing time-dependent coefficients that evolve with diffusion steps and spatial locations. Face-MoGLE provides a powerful and flexible solution for high-quality, controllable face generation, with strong potential in generative modeling and security applications. Extensive experiments demonstrate its effectiveness in multimodal and monomodal face generation settings and its robust zero-shot generalization capability. Project page is available at https://github.com/XavierJiezou/Face-MoGLE.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 58 of its 77 references that are in the map. First posted on arXiv 30 Aug 2025.

Builds on

58 of 77 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation18 Oct
  2. 2024One-Step Image Translation with Text-to-Image Models18 Mar
  3. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models13 Aug
  4. 2023Collaborative Diffusion for Multi-Modal Face Generation and Editing20 Apr
  5. 2023DCFace: Synthetic Face Generation with Dual Condition Diffusion Model14 Apr
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  7. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  9. 2022clip2latent: Text driven sampling of a pre-trained StyleGAN using denoising diffusion and CLIP5 Oct

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2020Denoising Diffusion Implicit Modelscited by 9,781
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2022Scalable Diffusion Models with Transformerscited by 6,054
37 more
  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  2. 2022Classifier-Free Diffusion Guidancecited by 5,847
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  11. 2018Demystifying MMD GANscited by 1,293
  12. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  13. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  14. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  15. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  16. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 628
  17. 2020GANSpace: Discovering Interpretable GAN Controlscited by 614
  18. 2020Plug-and-Play Image Restoration with Deep Denoiser Priorcited by 470
  19. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 340
  20. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 310
  21. 2022Splicing ViT Features for Semantic Appearance Transfercited by 209
  22. 2020Deep Learning-Based Human Pose Estimation: A Surveycited by 207
  23. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 195
  24. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 187
  25. 2022MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Videocited by 159
  26. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 133
  27. 2022Ego-Body Pose Estimation via Ego-Head Pose Estimationcited by 94
  28. 2019Local Class-Specific and Global Image-Level Generative Adversarial Networks for Semantic-Guided Scene Generationcited by 75
  29. 2021Diverse Semantic Image Synthesis via Probability Distribution Modelingcited by 49
  30. 2023Rethinking the Up-Sampling Operations in CNN-based Generative Network for Generalizable Deepfake Detectioncited by 41
  31. 2021E2Style: Improve the Efficiency and Effectiveness of StyleGAN Inversioncited by 33
  32. 2024SegFace: Face Segmentation of Long-Tail Classescited by 18
  33. 2024Diffusion-driven GAN Inversion for Multi-Modal Face Image Generationcited by 12
  34. 2023Learning to Generate Semantic Layouts for Higher Text-Image Correspondence in Text-to-Image Synthesiscited by 11
  35. 2024Fast Text-to-3D-Aware Face Generation and Manipulation via Direct Cross-modal Mapping and Geometric Regularizationcited by 8
  36. 2024Wavelet-Driven Generalizable Framework for Deepfake Face Forgery Detectioncited by 8
  37. 2024MM2Latent: Text-to-facial image generation and editing in GANs with multimodal assistancecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?cited by 6
  2. 2026PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Datasetcited by 1
  3. 2026Pixel-Space Diffusion Transformerscited by 1
  4. 2026Harnessing CLIP and DINO: An Uncertainty-Aware Cascaded Fusion Network for Generalizable Deepfake Image Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ExpertGen: Training-Free Expert Guidance for Controllable Text-to-Face Generation
  2. 2026MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation
  3. 2024Controllable Face Synthesis with Semantic Latent Diffusion Models
  4. 2025Bringing Diversity from Diffusion Models to Semantic-Guided Face Asset Generation
  5. 2025Multivariate Diffusion Transformer with Decoupled Attention for High-Fidelity Mask-Text Collaborative Facial Generation
  6. 2023Conditioning Diffusion Models via Attributes and Semantic Masks for Face Generation
  7. 2025Enhancing Diffusion Face Generation with Contrastive Embeddings and SegFormer Guidance
  8. 2024FilterPrompt: A Simple yet Efficient Approach to Guide Image Appearance Transfer in Diffusion Models