# MRecGen: Multimodal Appropriate Reaction Generator

arXiv: https://arxiv.org/abs/2307.02609 · HTML: https://mlsanity.com/paper/2307.02609/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 5 Jul 2023.

## Builds on (most cited first, 13 of 13)

- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022)
- [Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion](https://mlsanity.com/paper/2204.08451/) (2022)
- [Gesticulator: A framework for semantically-aware speech-driven gesture generation](https://mlsanity.com/paper/2001.09326/) (2020)
- [BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Prediction](https://mlsanity.com/paper/2211.14304/) (2022)
- [Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings](https://mlsanity.com/paper/2006.09888/) (2020)
- [Multiple Appropriate Facial Reaction Generation in Dyadic Interaction Settings: What, Why and How?](https://mlsanity.com/paper/2302.06514/) (2023)
- [Reversible Graph Neural Network-based Reaction Distribution Learning for Multiple Appropriate Facial Reactions Generation](https://mlsanity.com/paper/2305.15270/) (2023)
- [REACT2023: the first Multi-modal Multiple Appropriate Facial Reaction Generation Challenge](https://mlsanity.com/paper/2306.06583/) (2023)

## Built on by (most cited first, 2 of 2)

- [REACT 2024: the Second Multiple Appropriate Facial Reaction Generation Challenge](https://mlsanity.com/paper/2401.05166/) (2024)
- [ARIG: Autoregressive Interactive Head Generation for Real-time Conversations](https://mlsanity.com/paper/2507.00472/) (2025)

## Abstract (arXiv)

Verbal and non-verbal human reaction generation is a challenging task, as different reactions could be appropriate for responding to the same behaviour. This paper proposes the first multiple and multimodal (verbal and nonverbal) appropriate human reaction generation framework that can generate appropriate and realistic human-style reactions (displayed in the form of synchronised text, audio and video streams) in response to an input user behaviour. This novel technique can be applied to various human-computer interaction scenarios by generating appropriate virtual agent/robot behaviours. Our demo is available at urlhttps://github.com/SSYSteve/MRecGen.