# Every Smile is Unique: Landmark-Guided Diverse Smile Generation

arXiv: https://arxiv.org/abs/1802.01873 · HTML: https://mlsanity.com/paper/1802.01873/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2018.

## Built on by (most cited first, 23 of 23)

- [Motion Representations for Articulated Animation](https://mlsanity.com/paper/2104.11280/) (2021)
- [Image-to-Image Translation: Methods and Applications](https://mlsanity.com/paper/2101.08629/) (2021)
- [Talk-to-Edit: Fine-Grained Facial Editing via Dialog](https://mlsanity.com/paper/2109.04425/) (2021)
- [Make-A-Story: Visual Memory Conditioned Consistent Story Generation](https://mlsanity.com/paper/2211.13319/) (2022)
- [Detecting and Recovering Sequential DeepFake Manipulation](https://mlsanity.com/paper/2207.02204/) (2022)
- [Robust Differentiable SVD](https://mlsanity.com/paper/2104.03821/) (2021)
- [Sparse to Dense Dynamic 3D Facial Expression Generation](https://mlsanity.com/paper/2105.07463/) (2021)
- [Interactive Face Video Coding: A Generative Compression Framework](https://mlsanity.com/paper/2302.09919/) (2023)
- [4D Facial Expression Diffusion Model](https://mlsanity.com/paper/2303.16611/) (2023)
- [Total Generate: Cycle in Cycle Generative Adversarial Networks for Generating Human Faces, Hands, Bodies, and Natural Scenes](https://mlsanity.com/paper/2106.10876/) (2021)
- [Conditional Temporal Variational AutoEncoder for Action Video Prediction](https://mlsanity.com/paper/2108.05658/) (2021)
- [Continuous Piecewise-Affine Based Motion Model for Image Animation](https://mlsanity.com/paper/2401.09146/) (2024)
- [Generating Multiple 4D Expression Transitions by Learning Face Landmark Trajectories](https://mlsanity.com/paper/2208.00050/) (2022)
- [Facial Expression Translation using Landmark Guided GANs](https://mlsanity.com/paper/2209.02136/) (2022)
- [Facial Expression Video Generation Based-On Spatio-temporal Convolutional GAN: FEV-GAN](https://mlsanity.com/paper/2210.11182/) (2022)
- [Robust Sequential DeepFake Detection](https://mlsanity.com/paper/2309.14991/) (2023)
- [Differential Motion Evolution for Fine-Grained Motion Deformation in Unsupervised Image Animation](https://mlsanity.com/paper/2110.04658/) (2021)
- [A Review of Affective Generation Models](https://mlsanity.com/paper/2202.10763/) (2022)
- [Editable-DeepSC: Reliable Cross-Modal Semantic Communications for Facial Editing](https://mlsanity.com/paper/2411.15702/) (2024)
- [GroupDiff: Diffusion-based Group Portrait Editing](https://mlsanity.com/paper/2409.14379/) (2024)
- [Key-point Guided Deformable Image Manipulation Using Diffusion Model](https://mlsanity.com/paper/2401.08178/) (2024)
- [Learning to Generate Context-Sensitive Backchannel Smiles for Embodied AI Agents with Applications in Mental Health Dialogues](https://mlsanity.com/paper/2402.08837/) (2024)
- [GADS: A Super Lightweight Model for Head Pose Estimation](https://mlsanity.com/paper/2504.15751/) (2025)

## Abstract (arXiv)

Each smile is unique: one person surely smiles in different ways (e.g., closing/opening the eyes or mouth). Given one input image of a neutral face, can we generate multiple smile videos with distinctive characteristics? To tackle this one-to-many video generation problem, we propose a novel deep learning architecture named Conditional Multi-Mode Network (CMM-Net). To better encode the dynamics of facial expressions, CMM-Net explicitly exploits facial landmarks for generating smile sequences. Specifically, a variational auto-encoder is used to learn a facial landmark embedding. This single embedding is then exploited by a conditional recurrent network which generates a landmark embedding sequence conditioned on a specific expression (e.g., spontaneous smile). Next, the generated landmark embeddings are fed into a multi-mode recurrent landmark generator, producing a set of landmark sequences still associated to the given smile class but clearly distinct from each other. Finally, these landmark sequences are translated into face videos. Our experimental results demonstrate the effectiveness of our CMM-Net in generating realistic videos of multiple smile expressions.