# Reversible Graph Neural Network-based Reaction Distribution Learning for Multiple Appropriate Facial Reactions Generation

arXiv: https://arxiv.org/abs/2305.15270 · HTML: https://mlsanity.com/paper/2305.15270/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 57 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Invertible Residual Networks](https://mlsanity.com/paper/1811.00995/) (2018)
- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [The Lipschitz Constant of Self-Attention](https://mlsanity.com/paper/2006.04710/) (2020)
- [GANimation: Anatomically-aware Facial Animation from a Single Image](https://mlsanity.com/paper/1807.09251/) (2018)
- [Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion](https://mlsanity.com/paper/2204.08451/) (2022)
- [Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition](https://mlsanity.com/paper/2205.01782/) (2022)
- [Responsive Listening Head Generation: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2112.13548/) (2021)
- [Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning](https://mlsanity.com/paper/2108.00262/) (2021)
- [Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings](https://mlsanity.com/paper/2006.09888/) (2020)
- [Affective Faces for Goal-Driven Dyadic Communication](https://mlsanity.com/paper/2301.10939/) (2023)
- [Context-Aware Personality Inference in Dyadic Scenarios: Introducing the UDIVA Dataset](https://mlsanity.com/paper/2012.14259/) (2020)
- [Dynamic Facial Expression Generation on Hilbert Hypersphere with Conditional Wasserstein Generative Adversarial Nets](https://mlsanity.com/paper/1907.10087/) (2019)
- [Sparse to Dense Dynamic 3D Facial Expression Generation](https://mlsanity.com/paper/2105.07463/) (2021)
- [Multiple Appropriate Facial Reaction Generation in Dyadic Interaction Settings: What, Why and How?](https://mlsanity.com/paper/2302.06514/) (2023)
- [Interactive Generative Adversarial Networks for Facial Expression Generation in Dyadic Interactions](https://mlsanity.com/paper/1801.09092/) (2018)
- [GRATIS: Deep Learning Graph Representation with Task-specific Topology and Multi-dimensional Edge Features](https://mlsanity.com/paper/2211.12482/) (2022)
- [Expression Conditional GAN for Facial Expression-to-Expression Translation](https://mlsanity.com/paper/1905.05416/) (2019)
- [Bipartite Graph Reasoning GANs for Person Pose and Facial Image Synthesis](https://mlsanity.com/paper/2211.06719/) (2022)
- [AMII: Adaptive Multimodal Inter-personal and Intra-personal Model for Adapted Behavior Synthesis](https://mlsanity.com/paper/2305.11310/) (2023)
- [Facial Expression Translation using Landmark Guided GANs](https://mlsanity.com/paper/2209.02136/) (2022)

## Built on by (most cited first, 11 of 11)

- [ReactFace: Online Multiple Appropriate Facial Reaction Generation in Dyadic Interactions](https://mlsanity.com/paper/2305.15748/) (2023)
- [REACT 2024: the Second Multiple Appropriate Facial Reaction Generation Challenge](https://mlsanity.com/paper/2401.05166/) (2024)
- [REACT 2025: the Third Multiple Appropriate Facial Reaction Generation Challenge](https://mlsanity.com/paper/2505.17223/) (2025)
- [REACT2023: the first Multi-modal Multiple Appropriate Facial Reaction Generation Challenge](https://mlsanity.com/paper/2306.06583/) (2023)
- [Robust Facial Reactions Generation: An Emotion-Aware Framework with Modality Compensation](https://mlsanity.com/paper/2407.15798/) (2024)
- [MRecGen: Multimodal Appropriate Reaction Generator](https://mlsanity.com/paper/2307.02609/) (2023)
- [ReactDiff: Latent Diffusion for Facial Reaction Generation](https://mlsanity.com/paper/2505.14151/) (2025)
- [REACT 2026: The Fourth Multiple Appropriate Facial Reaction Generation Challenge: Personalised MAFRG and Appropriate EEG Reaction Prediction](https://mlsanity.com/paper/2606.07935/) (2026)
- [Conversational Human Audio-visual Talking Dialogue Generation](https://mlsanity.com/paper/2607.02799/) (2026)
- [ResDiffFRG: Residual Diffusion for Multiple Appropriate Facial Reaction Generation](https://mlsanity.com/paper/2609.33749/) (2026)
- [Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generation](https://mlsanity.com/paper/2609.34419/) (2026)

## Abstract (arXiv)

Generating facial reactions in a human-human dyadic interaction is complex and highly dependent on the context since more than one facial reactions can be appropriate for the speaker's behaviour. This has challenged existing machine learning (ML) methods, whose training strategies enforce models to reproduce a specific (not multiple) facial reaction from each input speaker behaviour. This paper proposes the first multiple appropriate facial reaction generation framework that re-formulates the one-to-many mapping facial reaction generation problem as a one-to-one mapping problem. This means that we approach this problem by considering the generation of a distribution of the listener's appropriate facial reactions instead of multiple different appropriate facial reactions, i.e., 'many' appropriate facial reaction labels are summarised as 'one' distribution label during training. Our model consists of a perceptual processor, a cognitive processor, and a motor processor. The motor processor is implemented with a novel Reversible Multi-dimensional Edge Graph Neural Network (REGNN). This allows us to obtain a distribution of appropriate real facial reactions during the training process, enabling the cognitive processor to be trained to predict the appropriate facial reaction distribution. At the inference stage, the REGNN decodes an appropriate facial reaction by using this distribution as input. Experimental results demonstrate that our approach outperforms existing models in generating more appropriate, realistic, and synchronized facial reactions. The improved performance is largely attributed to the proposed appropriate facial reaction distribution learning strategy and the use of a REGNN. The code is available at https://github.com/TongXu-05/REGNN-Multiple-Appropriate-Facial-Reaction-Generation.