# Joker: Conditional 3D Head Synthesis with Extreme Facial Expressions

arXiv: https://arxiv.org/abs/2410.16395 · HTML: https://mlsanity.com/paper/2410.16395/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 83 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 30 of 62)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [CAT3D: Create Anything in 3D with Multi-View Diffusion Models](https://mlsanity.com/paper/2405.10314/) (2024)
- [ReconFusion: 3D Reconstruction with Diffusion Priors](https://mlsanity.com/paper/2312.02981/) (2023)
- [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019)
- [ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2312.02201/) (2023)
- [Motion Representations for Articulated Animation](https://mlsanity.com/paper/2104.11280/) (2021)
- [EMOCA: Emotion Driven Monocular Face Capture and Animation](https://mlsanity.com/paper/2204.11312/) (2022)
- [CelebV-HQ: A Large-Scale Video Facial Attributes Dataset](https://mlsanity.com/paper/2207.12393/) (2022)
- [NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads](https://mlsanity.com/paper/2305.03027/) (2023)
- [X2Face: A network for controlling face generation by using images, audio, and pose codes](https://mlsanity.com/paper/1807.10550/) (2018)
- [Depth-Aware Generative Adversarial Network for Talking Head Video Generation](https://mlsanity.com/paper/2203.06605/) (2022)
- [Deep Appearance Models for Face Rendering](https://mlsanity.com/paper/1808.00362/) (2018)
- [MegaPortraits: One-shot Megapixel Neural Head Avatars](https://mlsanity.com/paper/2207.07621/) (2022)
- [MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition](https://mlsanity.com/paper/2011.11961/) (2020)

## Built on by (most cited first, 18 of 18)

- [CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Models](https://mlsanity.com/paper/2412.12093/) (2024)
- [Pixel3DMM: Versatile Screen-Space Priors for Single-Image 3D Face Reconstruction](https://mlsanity.com/paper/2505.00615/) (2025)
- [DiffPortrait360: Consistent Portrait Diffusion for 360 View Synthesis](https://mlsanity.com/paper/2503.15667/) (2025)
- [Controlling Avatar Diffusion with Learnable Gaussian Embedding](https://mlsanity.com/paper/2503.15809/) (2025)
- [SpinMeRound: Consistent Multi-View Identity Generation Using Diffusion Models](https://mlsanity.com/paper/2504.10716/) (2025)
- [Bringing Your Portrait to 3D Presence](https://mlsanity.com/paper/2511.22553/) (2025)
- [Text-based Animatable 3D Avatars with Morphable Model Alignment](https://mlsanity.com/paper/2504.15835/) (2025)
- [GNM Head: A Generative aNthropometric Model of the human head](https://mlsanity.com/paper/2607.23687/) (2026)
- [FactorPortrait: Controllable Portrait Animation via Disentangled Expression, Pose, and Viewpoint](https://mlsanity.com/paper/2512.11645/) (2025)
- [Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image](https://mlsanity.com/paper/2512.17773/) (2025)
- [Feed-forward Gaussian Registration for Head Avatar Creation and Editing](https://mlsanity.com/paper/2603.15811/) (2026)
- [GenLCA: 3D Diffusion for Full-Body Avatars from In-the-Wild Videos](https://mlsanity.com/paper/2604.07273/) (2026)
- [EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions](https://mlsanity.com/paper/2607.02674/) (2026)
- [GRMM: Real-Time High-Fidelity Gaussian Morphable Head Model with Learned Residuals](https://mlsanity.com/paper/2509.02141/) (2025)
- [CrowdGaussian: Reconstructing High-Fidelity 3D Gaussians for Human Crowd from a Single Image](https://mlsanity.com/paper/2603.17779/) (2026)
- [GSwap: Realistic Head Swapping with Dynamic Neural Gaussian Field](https://mlsanity.com/paper/2603.23168/) (2026)
- [AHS: Adaptive Head Synthesis via Synthetic Data Augmentations](https://mlsanity.com/paper/2604.15857/) (2026)
- [ViDS: Video Diffusion Shader using 3D Face Tracking](https://mlsanity.com/paper/2607.24124/) (2026)

## Abstract (arXiv)

We introduce Joker, a new method for the conditional synthesis of 3D human heads with extreme expressions. Given a single reference image of a person, we synthesize a volumetric human head with the reference identity and a new expression. We offer control over the expression via a 3D morphable model (3DMM) and textual inputs. This multi-modal conditioning signal is essential since 3DMMs alone fail to define subtle emotional changes and extreme expressions, including those involving the mouth cavity and tongue articulation. Our method is built upon a 2D diffusion-based prior that generalizes well to out-of-domain samples, such as sculptures, heavy makeup, and paintings while achieving high levels of expressiveness. To improve view consistency, we propose a new 3D distillation technique that converts predictions of our 2D prior into a neural radiance field (NeRF). Both the 2D prior and our distillation technique produce state-of-the-art results, which are confirmed by our extensive evaluations. Also, to the best of our knowledge, our method is the first to achieve view-consistent extreme tongue articulation.