mlsanityData to 7 Oct 2026

Photorealistic and Identity-Preserving Image-Based Emotion Manipulation with Latent Diffusion Models

arXiv 2308.03183 ↗First posted 6 Aug 2023cs.CVcited by 5

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

In this paper, we investigate the emotion manipulation capabilities of diffusion models with "in-the-wild" images, a rather unexplored application area relative to the vast and rapidly growing literature for image-to-image translation tasks. Our proposed method encapsulates several pieces of prior work, with the most important being Latent Diffusion models and text-driven manipulation with CLIP latents. We conduct extensive qualitative and quantitative evaluations on AffectNet, demonstrating the superiority of our approach in terms of image quality and realism, while achieving competitive results relative to emotion translation compared to a variety of GAN-based counterparts. Code is released as a publicly available repo.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 39 of its 52 references that are in the map. First posted on arXiv 6 Aug 2023.

Builds on

39 of 52 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2017Squeeze-and-Excitation Networkscited by 4,763
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
27 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  3. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  4. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,095
  5. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  6. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  7. 2018CosFace: Large Margin Cosine Loss for Deep Face Recognitioncited by 968
  8. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  9. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  10. 2023Zero-shot Image-to-Image Translationcited by 503
  11. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  12. 2017AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wildcited by 490
  13. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  14. 2022Diffusion models as plug-and-play priorscited by 230
  15. 2021Diffusion Models for Implicit Image Segmentation Ensemblescited by 198
  16. 2022EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equationscited by 172
  17. 2020Partial FC: Training 10 Million Identities on a Single Machinecited by 163
  18. 2018GANimation: Anatomically-aware Facial Animation from a Single Imagecited by 160
  19. 2016Learning in Implicit Generative Modelscited by 124
  20. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  21. 2022DiffFace: Diffusion-based Face Swapping with Facial Guidancecited by 77
  22. 2019ICface: Interpretable and Controllable Face Reenactment Using GANscited by 50
  23. 2017ExprGAN: Facial Expression Editing with Controllable Expression Intensitycited by 47
  24. 2021Facial expression and attributes recognition based on multi-task learning of lightweight neural networkscited by 40
  25. 2020FACEGAN: Facial Attribute Controllable rEenactment GANcited by 32
  26. 2020Facial Expression Editing with Continuous Emotion Labelscited by 6
  27. 2023StyO: Stylize Your Face in Only One-shotcited by 6

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 4.

5 papers.

  1. 2024MagicNaming: Consistent Identity Generation by Finding a "Name Space" in T2I Diffusion Modelscited by 5
  2. 2025Emotion Recognition and Generation: A Comprehensive Review of Face, Speech, and Text Modalitiescited by 2
  3. 2025GenEAva: Generating Cartoon Avatars with Fine-Grained Facial Expressions from Realistic Diffusion-based Facescited by 0
  4. 2025Gen-AFFECT: Generation of Avatar Fine-grained Facial Expressions with Consistent identiTycited by 0
  5. 2025BeyondFacial: Identity-Preserving Personalized Generation Beyond Facial Close-upscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Make Me Happier: Evoking Emotions Through Image Diffusion Models
  2. 2025MUSE: Manipulating Unified Framework for Synthesizing Emotions in Images via Test-Time Optimization
  3. 2023NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models
  4. 2025Towards Deeper Emotional Reflection: Crafting Affective Image Filters with Generative Priors
  5. 2022LDEdit: Towards Generalized Text Guided Image Manipulation via Latent Diffusion Models
  6. 2025Emotion-Director: Bridging Affective Shortcut in Emotion-Oriented Image Generation
  7. 2024A Review of Human Emotion Synthesis Based on Generative Technology
  8. 2026EmoScene: A Dual-space Dataset for Controllable Affective Image Generation