# ResDiffFRG: Residual Diffusion for Multiple Appropriate Facial Reaction Generation

arXiv: https://arxiv.org/abs/2609.33749 · HTML: https://mlsanity.com/paper/2609.33749/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Builds on (most cited first, 14 of 14)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion](https://mlsanity.com/paper/2204.08451/) (2022)
- [FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset](https://mlsanity.com/paper/2203.14057/) (2022)
- [Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition](https://mlsanity.com/paper/2205.01782/) (2022)
- [Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning](https://mlsanity.com/paper/2112.03763/) (2021)
- [Multiple Appropriate Facial Reaction Generation in Dyadic Interaction Settings: What, Why and How?](https://mlsanity.com/paper/2302.06514/) (2023)
- [Interactive Generative Adversarial Networks for Facial Expression Generation in Dyadic Interactions](https://mlsanity.com/paper/1801.09092/) (2018)
- [Reversible Graph Neural Network-based Reaction Distribution Learning for Multiple Appropriate Facial Reactions Generation](https://mlsanity.com/paper/2305.15270/) (2023)
- [REACT 2025: the Third Multiple Appropriate Facial Reaction Generation Challenge](https://mlsanity.com/paper/2505.17223/) (2025)
- [AMII: Adaptive Multimodal Inter-personal and Intra-personal Model for Adapted Behavior Synthesis](https://mlsanity.com/paper/2305.11310/) (2023)
- [ReactDiff: Fundamental Multiple Appropriate Facial Reaction Diffusion Model](https://mlsanity.com/paper/2510.04712/) (2025)
- [ReactDiff: Latent Diffusion for Facial Reaction Generation](https://mlsanity.com/paper/2505.14151/) (2025)

## Abstract (arXiv)

In dyadic human speaker-listener conversations, the listener's facial reactions allows the speaker to accurately perceive the listener's emotional states. Since human facial reactions are non-deterministic, the ability to generate multiple appropriate human-like facial reactions is crucial for realistic human-agent interactions. Although diffusion models are naturally suited to such one-to-many generation, existing diffusion-based Multiple Appropriate Facial Reaction Generation (MAFRG) methods attempt to denoise random Gaussian initialisations directly into multiple appropriate facial reactions (AFRs). These random initialisations are usually not well-aligned with the target listener facial reaction, which requires complex denoising trajectories from these initialisations, and subsequently creates substantial opportunities for deviations away from the range of trajectories leading to appropriate AFRs. Given the inherent mimicry between the human listener's and speaker's facial behaviours, we address the above denoising trajectory issue by leveraging this strong prior. Specifically, we propose ResDiffFRG, a novel diffusion-based MAFRG framework that explicitly anchors the diffusion process to the speaker behaviour by defining its diffusion target as the residual between the speaker anchor and an AFR. The denoiser only needs to model the comparatively small, reaction-specific residual needed to transform this anchor into an AFR, rather than reconstructing the complete reaction from an unstructured state. Extensive experiments show that ResDiffFRG achieves large improvements in correlation-based appropriateness over existing methods. Our denoising trajectory analysis showed that even at the start of the denoising trajectory, ResDiffFRG already achieves a higher facial-reaction correlation score than the Gaussian Diffusion baseline does after completing 60% of its denoising trajectory.