# EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models

arXiv: https://arxiv.org/abs/2401.04608 · HTML: https://mlsanity.com/paper/2401.04608/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 30 of its 61 references that are in the map.
- First posted on arXiv 9 Jan 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation](https://mlsanity.com/paper/2302.13848/) (2023)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Models](https://mlsanity.com/paper/2308.07863/) (2023)
- [EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes](https://mlsanity.com/paper/2307.07961/) (2023)
- [Pastiche Master: Exemplar-Based High-Resolution Portrait Style Transfer](https://mlsanity.com/paper/2203.13248/) (2022)
- [Text to Image Generation with Semantic-Spatial Aware GAN](https://mlsanity.com/paper/2104.00567/) (2021)
- [Diverse Semantic Image Synthesis via Probability Distribution Modeling](https://mlsanity.com/paper/2103.06878/) (2021)
- [Learning Multi-level Deep Representations for Image Emotion Classification](https://mlsanity.com/paper/1611.07145/) (2016)
- [Stimuli-Aware Visual Emotion Analysis](https://mlsanity.com/paper/2109.01812/) (2021)
- [SOLVER: Scene-Object Interrelated Visual Emotion Reasoning Network](https://mlsanity.com/paper/2110.12334/) (2021)
- [NoisyTwins: Class-Consistent and Diverse Image Generation through StyleGANs](https://mlsanity.com/paper/2304.05866/) (2023)
- [Image Sentiment Transfer](https://mlsanity.com/paper/2006.11337/) (2020)

## Built on by (most cited first, 30 of 37)

- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [EmoEdit: Evoking Emotions through Image Manipulation](https://mlsanity.com/paper/2405.12661/) (2024)
- [EmotiCrafter: Text-to-Emotional-Image Generation based on Valence-Arousal Model](https://mlsanity.com/paper/2501.05710/) (2025)
- [UniEmo: Unifying Emotional Understanding and Generation with Learnable Expert Queries](https://mlsanity.com/paper/2507.23372/) (2025)
- [Knowledge-Aligned Counterfactual-Enhancement Diffusion Perception for Unsupervised Cross-Domain Visual Emotion Recognition](https://mlsanity.com/paper/2505.19694/) (2025)
- [Affective Image Editing: Shaping Emotional Factors via Text Descriptions](https://mlsanity.com/paper/2505.18699/) (2025)
- [MEMO-Bench: A Multiple Benchmark for Text-to-Image and Multimodal Large Language Models on Human Emotion Analysis](https://mlsanity.com/paper/2411.11235/) (2024)
- [EmoVerse: A MLLMs-Driven Emotion Representation Dataset for Interpretable Visual Emotion Analysis](https://mlsanity.com/paper/2511.12554/) (2025)
- [EmoFeedback2: Reinforcement of Continuous Emotional Image Generation via LVLM-based Reward and Textual Feedback](https://mlsanity.com/paper/2511.19982/) (2025)
- [HERO: Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuning](https://mlsanity.com/paper/2410.05116/) (2024)
- [EmoCtrl: Controllable Emotional Image Content Generation](https://mlsanity.com/paper/2512.22437/) (2025)
- [EmoKGEdit: Training-free Affective Injection via Visual Cue Transformation](https://mlsanity.com/paper/2601.12326/) (2026)
- [Improved Emotional Alignment of AI and Humans: Human Ratings of Emotions Expressed by Stable Diffusion v1, DALL-E 2, and DALL-E 3](https://mlsanity.com/paper/2405.18510/) (2024)
- [EmoAgent: A Multi-Agent Framework for Diverse Affective Image Manipulation](https://mlsanity.com/paper/2503.11290/) (2025)
- [MUSE: Manipulating Unified Framework for Synthesizing Emotions in Images via Test-Time Optimization](https://mlsanity.com/paper/2511.21051/) (2025)
- [EmoStyle: Emotion-Driven Image Stylization](https://mlsanity.com/paper/2512.05478/) (2025)
- [Emotion-Director: Bridging Affective Shortcut in Emotion-Oriented Image Generation](https://mlsanity.com/paper/2512.19479/) (2025)
- [eMotions: A Large-Scale Dataset and Audio-Visual Fusion Network for Emotion Analysis in Short-form Videos](https://mlsanity.com/paper/2508.06902/) (2025)
- [Neural Scene Designer: Self-Styled Semantic Image Manipulation](https://mlsanity.com/paper/2509.01405/) (2025)
- [AIM-Bench: Benchmarking and Improving Affective Image Manipulation via Fine-Grained Hierarchical Control](https://mlsanity.com/paper/2604.10454/) (2026)
- [What Can I Edit? Open-Ended Strategy Discovery and the Emotion Editability Landscape](https://mlsanity.com/paper/2607.23920/) (2026)
- [Who is a Better Talker: Subjective and Objective Quality Assessment for AI-Generated Talking Heads](https://mlsanity.com/paper/2507.23343/) (2025)
- [EmoSpace: Immersive Affective Image Generation Guided by Fine-Grained Emotion Prototypes](https://mlsanity.com/paper/2602.11658/) (2026)
- [CogBlender: Towards Continuous Cognitive Intervention in Text-to-Image Generation](https://mlsanity.com/paper/2603.09286/) (2026)
- [EmoStory: Emotion-Aware Story Generation](https://mlsanity.com/paper/2603.10349/) (2026)
- [MER-Bench: A Comprehensive Benchmark for Multimodal Meme Reappraisal](https://mlsanity.com/paper/2603.15020/) (2026)
- [EmoScene: A Dual-space Dataset for Controllable Affective Image Generation](https://mlsanity.com/paper/2604.00933/) (2026)
- [EPIG: Emotion-Based Prompting for Personalised Image Generation](https://mlsanity.com/paper/2606.13247/) (2026)
- [Visually grounded emotion regulation via diffusion models and user-driven reappraisal](https://mlsanity.com/paper/2507.10861/) (2025)
- [Security Risk of Misalignment between Text and Image in Multi-modal Model](https://mlsanity.com/paper/2510.26105/) (2025)

## Abstract (arXiv)

Recent years have witnessed remarkable progress in image generation task, where users can create visually astonishing images with high-quality. However, existing text-to-image diffusion models are proficient in generating concrete concepts (dogs) but encounter challenges with more abstract ones (emotions). Several efforts have been made to modify image emotions with color and style adjustments, facing limitations in effectively conveying emotions with fixed image contents. In this work, we introduce Emotional Image Content Generation (EICG), a new task to generate semantic-clear and emotion-faithful images given emotion categories. Specifically, we propose an emotion space and construct a mapping network to align it with the powerful Contrastive Language-Image Pre-training (CLIP) space, providing a concrete interpretation of abstract emotions. Attribute loss and emotion confidence are further proposed to ensure the semantic diversity and emotion fidelity of the generated images. Our method outperforms the state-of-the-art text-to-image approaches both quantitatively and qualitatively, where we derive three custom metrics, i.e., emotion accuracy, semantic clarity and semantic diversity. In addition to generation, our method can help emotion understanding and inspire emotional art design.