mlsanityData to 7 Oct 2026

Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion

arXiv 2204.08451 ↗First posted 18 Apr 2022cs.CVcited by 125

Not on the map. Why?

Abstract (arXiv)

We present a framework for modeling interactional communication in dyadic conversations: given multimodal inputs of a speaker, we autoregressively output multiple possibilities of corresponding listener motion. We combine the motion and speech audio of the speaker using a motion-audio cross attention transformer. Furthermore, we enable non-deterministic prediction by learning a discrete latent representation of realistic listener motion with a novel motion-encoding VQ-VAE. Our method organically captures the multimodal and non-deterministic nature of nonverbal dyadic interactions. Moreover, it produces realistic 3D listener facial motion synchronous with the speaker (see video). We demonstrate that our method outperforms baselines qualitatively and quantitatively via a rich suite of experiments. To facilitate this line of research, we introduce a novel and large in-the-wild dataset of dyadic conversations. Code, data, and videos available at https://evonneng.github.io/learning2listen/.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Builds on 24 of its 70 references that are in the map. First posted on arXiv 18 Apr 2022.

Builds on

24 of 70 references.

References · 24

  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  6. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  7. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  8. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  9. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  10. 2018Video-to-Video Synthesiscited by 388
  11. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  12. 2020Long-term Human Motion Prediction with Scene Contextcited by 191
12 more
  1. 2019Learning Individual Styles of Conversational Gesturecited by 190
  2. 2019Generating 3D People in Scenes without Peoplecited by 127
  3. 2019Structured Prediction Helps 3D Human Motion Modellingcited by 96
  4. 2020Learning to Generate Diverse Dance Motions with Transformercited by 85
  5. 2015A Century of Portraits: A Visual Historical Record of American High School Yearbookscited by 54
  6. 2019Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interactioncited by 48
  7. 2019MoGlow: Probabilistic and controllable motion synthesis using normalising flowscited by 47
  8. 2020Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settingscited by 36
  9. 2020Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamicscited by 34
  10. 2019To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversationscited by 26
  11. 2018Interactive Generative Adversarial Networks for Facial Expression Generation in Dyadic Interactionscited by 15
  12. 2018A Face-to-Face Neural Conversation Modelcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 27; 2024: 28; 2025: 31; 2026: 35.

125 papers.

Sort:

By area:

  1. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  2. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  3. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  4. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  5. 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82
  6. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 77
  7. 2023ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactionscited by 62
  8. 2022Audio-Driven Co-Speech Gesture Video Generationcited by 54
  9. 2023Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environmentscited by 51
  10. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  11. 2023EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generationcited by 40
  12. 2023Can Language Models Learn to Listen?cited by 40
38 more
  1. 2024INFP: Audio-Driven Interactive Head Generation in Dyadic Conversationscited by 40
  2. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  3. 2024FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantizationcited by 39
  4. 2023Affective Faces for Goal-Driven Dyadic Communicationcited by 34
  5. 2024Dyadic Interaction Modeling for Social Behavior Generationcited by 32
  6. 2025DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversationscited by 31
  7. 2023FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learningcited by 29
  8. 2024Towards Variable and Coordinated Holistic Co-Speech Motion Generationcited by 29
  9. 2024MultiTalk: Enhancing 3D Talking Head Generation Across Languages with Multilingual Video Datasetcited by 28
  10. 2024Towards Open Domain Text-Driven Synthesis of Multi-Person Motionscited by 27
  11. 2025SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generationcited by 27
  12. 2024Say Anything with Any Stylecited by 26
  13. 2025StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatarscited by 24
  14. 2024CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generationcited by 22
  15. 2024Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questionscited by 21
  16. 2025DuetGen: Music Driven Two-Person Dance Generation via Hierarchical Masked Modelingcited by 21
  17. 2023Egocentric Auditory Attention Localization in Conversationscited by 18
  18. 2023ReactFace: Online Multiple Appropriate Facial Reaction Generation in Dyadic Interactionscited by 17
  19. 2023Emotional Listener Portrait: Neural Listener Head Generation with Emotioncited by 17
  20. 2026Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversationcited by 17
  21. 2023Multiple Appropriate Facial Reaction Generation in Dyadic Interaction Settings: What, Why and How?cited by 16
  22. 2024REACT 2024: the Second Multiple Appropriate Facial Reaction Generation Challengecited by 16
  23. 2025ARIG: Autoregressive Interactive Head Generation for Real-time Conversationscited by 16
  24. 2023Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applicationscited by 15
  25. 2024Beyond Talking -- Generating Holistic 3D Human Dyadic Motion for Communicationcited by 14
  26. 2025SocialGen: Modeling Multi-Human Social Interaction with Language Modelscited by 14
  27. 2024ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAEcited by 13
  28. 2022It Takes Two: Learning to Plan for Human-Robot Cooperative Carryingcited by 12
  29. 2024Text-centric Alignment for Multi-Modality Learningcited by 12
  30. 2024DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animationcited by 12
  31. 2023Reversible Graph Neural Network-based Reaction Distribution Learning for Multiple Appropriate Facial Reactions Generationcited by 11
  32. 2023AgentAvatar: Disentangling Planning, Driving and Rendering for Photorealistic Avatar Agentscited by 11
  33. 2025DiTaiListener: Controllable High Fidelity Listener Video Generation with Diffusioncited by 10
  34. 2025UniLS: End-to-End Audio-Driven Avatars for Unified Listening and Speakingcited by 10
  35. 2025REACT 2025: the Third Multiple Appropriate Facial Reaction Generation Challengecited by 9
  36. 2024GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformercited by 8
  37. 2024Synergy and Synchrony in Couple Dancescited by 8
  38. 2025Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agentscited by 8

50 most cited of 125 All 125 (JSON)