# To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations

arXiv: https://arxiv.org/abs/1910.02181 · HTML: https://mlsanity.com/paper/1910.02181/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2019.

## Built on by (most cited first, 26 of 26)

- [Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion](https://mlsanity.com/paper/2204.08451/) (2022)
- [From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations](https://mlsanity.com/paper/2401.01885/) (2024)
- [A Comprehensive Review of Data-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2301.05339/) (2023)
- [ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactions](https://mlsanity.com/paper/2311.17057/) (2023)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [Interaction Transformer for Human Reaction Generation](https://mlsanity.com/paper/2207.01685/) (2022)
- [Can Language Models Learn to Listen?](https://mlsanity.com/paper/2308.10897/) (2023)
- [Dyadic Interaction Modeling for Social Behavior Generation](https://mlsanity.com/paper/2403.09069/) (2024)
- [A Survey on Human Interaction Motion Generation](https://mlsanity.com/paper/2503.12763/) (2025)
- [Emotional Listener Portrait: Neural Listener Head Generation with Emotion](https://mlsanity.com/paper/2310.00068/) (2023)
- [Deep Person Generation: A Survey from the Perspective of Face, Pose and Cloth Synthesis](https://mlsanity.com/paper/2109.02081/) (2021)
- [Beyond Talking -- Generating Holistic 3D Human Dyadic Motion for Communication](https://mlsanity.com/paper/2403.19467/) (2024)
- [Didn't see that coming: a survey on non-verbal social human behavior forecasting](https://mlsanity.com/paper/2203.02480/) (2022)
- [AgentAvatar: Disentangling Planning, Driving and Rendering for Photorealistic Avatar Agents](https://mlsanity.com/paper/2311.17465/) (2023)
- [DiTaiListener: Controllable High Fidelity Listener Video Generation with Diffusion](https://mlsanity.com/paper/2504.04010/) (2025)
- [Comparison of Spatio-Temporal Models for Human Motion and Pose Forecasting in Face-to-Face Interaction Scenarios](https://mlsanity.com/paper/2203.03245/) (2022)
- [Zero-Shot Style Transfer for Gesture Animation driven by Text and Speech using Adversarial Disentanglement of Multimodal Style Encoding](https://mlsanity.com/paper/2208.01917/) (2022)
- [Social Processes: Self-Supervised Meta-Learning over Conversational Groups for Forecasting Nonverbal Social Cues](https://mlsanity.com/paper/2107.13576/) (2021)
- [Beyond the Sum: Unlocking AI Agents Potential Through Market Forces](https://mlsanity.com/paper/2501.10388/) (2024)
- [Latent Behavior Diffusion for Sequential Reaction Generation in Dyadic Setting](https://mlsanity.com/paper/2505.07901/) (2025)
- [SARAH: Spatially Aware Real-time Agentic Humans](https://mlsanity.com/paper/2602.18432/) (2026)
- [Why Did This Model Forecast This Future? Closed-Form Temporal Saliency Towards Causal Explanations of Probabilistic Forecasts](https://mlsanity.com/paper/2206.00679/) (2022)
- [A Probabilistic Model Of Interaction Dynamics for Dyadic Face-to-Face Settings](https://mlsanity.com/paper/2207.04566/) (2022)
- [ZS-MSTM: Zero-Shot Style Transfer for Gesture Animation driven by Text and Speech using Adversarial Disentanglement of Multimodal Style Encoding](https://mlsanity.com/paper/2305.12887/) (2023)
- [Decoding Psychological States Through Movement: Inferring Human Kinesic Functions with Application to Built Environments](https://mlsanity.com/paper/2601.17194/) (2026)
- [PhysiGen: Integrating Collision-Aware Physical Constraints for High-Fidelity Human-Human Interaction Generation](https://mlsanity.com/paper/2605.00517/) (2026)

## Abstract (arXiv)

Non verbal behaviours such as gestures, facial expressions, body posture, and para-linguistic cues have been shown to complement or clarify verbal messages. Hence to improve telepresence, in form of an avatar, it is important to model these behaviours, especially in dyadic interactions. Creating such personalized avatars not only requires to model intrapersonal dynamics between a avatar's speech and their body pose, but it also needs to model interpersonal dynamics with the interlocutor present in the conversation. In this paper, we introduce a neural architecture named Dyadic Residual-Attention Model (DRAM), which integrates intrapersonal (monadic) and interpersonal (dyadic) dynamics using selective attention to generate sequences of body pose conditioned on audio and body pose of the interlocutor and audio of the human operating the avatar. We evaluate our proposed model on dyadic conversational data consisting of pose and audio of both participants, confirming the importance of adaptive attention between monadic and dyadic dynamics when predicting avatar pose. We also conduct a user study to analyze judgments of human observers. Our results confirm that the generated body pose is more natural, models intrapersonal dynamics and interpersonal dynamics better than non-adaptive monadic/dyadic models.