mlsanityData to 7 Oct 2026

Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings

arXiv 2006.09888 ↗First posted 11 Jun 2020cs.CV cs.HC cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

To enable more natural face-to-face interactions, conversational agents need to adapt their behavior to their interlocutors. One key aspect of this is generation of appropriate non-verbal behavior for the agent, for example facial gestures, here defined as facial expressions and head movements. Most existing gesture-generating systems do not utilize multi-modal cues from the interlocutor when synthesizing non-verbal behavior. Those that do, typically use deterministic methods that risk producing repetitive and non-vivid motions. In this paper, we introduce a probabilistic method to synthesize interlocutor-aware facial gestures - represented by highly expressive FLAME parameters - in dyadic conversations. Our contributions are: a) a method for feature extraction from multi-party video and speech recordings, resulting in a representation that allows for independent control and manipulation of expression and speech articulation in a 3D avatar; b) an extension to MoGlow, a recent motion-synthesis method based on normalizing flows, to also take multi-modal signals from the interlocutor as input and subsequently output interlocutor-aware facial gestures; and c) a subjective evaluation assessing the use and relative importance of the input modalities. The results show that the model successfully leverages the input from the interlocutor to generate more appropriate behavior. Videos, data, and code available at: https://jonepatr.github.io/lets_face_it.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 19; 2024: 7; 2025: 3; 2026: 2.

36 papers.

Sort:

By area:

  1. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  2. 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 125
  3. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 77
  4. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  5. 2022The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generationcited by 40
  6. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  7. 2023Affective Faces for Goal-Driven Dyadic Communicationcited by 34
  8. 2024Dyadic Interaction Modeling for Social Behavior Generationcited by 32
  9. 2023FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learningcited by 29
  10. 2023Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022cited by 29
  11. 2023The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settingscited by 27
  12. 2023Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representationcited by 27
24 more
  1. 2023The Ethical Implications of Generative Audio Models: A Systematic Literature Reviewcited by 18
  2. 2023ReactFace: Online Multiple Appropriate Facial Reaction Generation in Dyadic Interactionscited by 17
  3. 2023Emotional Listener Portrait: Neural Listener Head Generation with Emotioncited by 17
  4. 2024Beyond Talking -- Generating Holistic 3D Human Dyadic Motion for Communicationcited by 14
  5. 2023Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesiscited by 13
  6. 2023Reversible Graph Neural Network-based Reaction Distribution Learning for Multiple Appropriate Facial Reactions Generationcited by 11
  7. 2025DiTaiListener: Controllable High Fidelity Listener Video Generation with Diffusioncited by 10
  8. 2022Zero-Shot Style Transfer for Gesture Animation driven by Text and Speech using Adversarial Disentanglement of Multimodal Style Encodingcited by 7
  9. 2023AMII: Adaptive Multimodal Inter-personal and Intra-personal Model for Adapted Behavior Synthesiscited by 6
  10. 2023Unified speech and gesture synthesis using flow matchingcited by 6
  11. 2023REACT2023: the first Multi-modal Multiple Appropriate Facial Reaction Generation Challengecited by 5
  12. 2023Towards the generation of synchronized and believable non-verbal facial behaviors of a talking virtual agentcited by 5
  13. 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5
  14. 2024Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesiscited by 3
  15. 2026PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interactioncited by 3
  16. 2023MRecGen: Multimodal Appropriate Reaction Generatorcited by 2
  17. 2024Allo-AVA: A Large-Scale Multimodal Conversational AI Dataset for Allocentric Avatar Gesture Animationcited by 2
  18. 2024How Video Meetings Change Your Expressioncited by 1
  19. 2025MAG: Multi-Modal Aligned Autoregressive Co-Speech Gesture Generation without Vector Quantizationcited by 1
  20. 2025Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmarkcited by 1
  21. 2022A Probabilistic Model Of Interaction Dynamics for Dyadic Face-to-Face Settingscited by 0
  22. 2023ZS-MSTM: Zero-Shot Style Transfer for Gesture Animation driven by Text and Speech using Adversarial Disentanglement of Multimodal Style Encodingcited by 0
  23. 2023Towards an architectural framework for intelligent virtual agents using probabilistic programmingcited by 0
  24. 2026LLM-Based Selection of Incongruent Verbal and Nonverbal Behavior for Virtual Humanscited by 0