# DyaPlex: Full-Duplex Speech-Motion Model for Dyadic Interaction

arXiv: https://arxiv.org/abs/2606.03874 · HTML: https://mlsanity.com/paper/2606.03874/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 40 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Builds on (most cited first, 30 of 37)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions](https://mlsanity.com/paper/2304.05684/) (2023)
- [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020)
- [Generating Holistic 3D Human Motion from Speech](https://mlsanity.com/paper/2212.04420/) (2022)
- [BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis](https://mlsanity.com/paper/2203.05297/) (2022)
- [Panoptic Studio: A Massively Multiview System for Social Interaction Capture](https://mlsanity.com/paper/1612.03153/) (2016)
- [Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2303.09119/) (2023)
- [Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion](https://mlsanity.com/paper/2204.08451/) (2022)
- [EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling](https://mlsanity.com/paper/2401.00374/) (2023)
- [Learning Speech-driven 3D Conversational Gestures from Video](https://mlsanity.com/paper/2102.06837/) (2021)
- [From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations](https://mlsanity.com/paper/2401.01885/) (2024)
- [PoseGPT: Quantization-based 3D Human Motion Generation and Forecasting](https://mlsanity.com/paper/2210.10542/) (2022)
- [PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models](https://mlsanity.com/paper/2602.06053/) (2026)
- [ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactions](https://mlsanity.com/paper/2311.17057/) (2023)
- [ReGenNet: Towards Human Action-Reaction Synthesis](https://mlsanity.com/paper/2403.11882/) (2024)
- [Visual Speech-Aware Perceptual 3D Facial Expression Reconstruction from Videos](https://mlsanity.com/paper/2207.11094/) (2022)
- [GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling](https://mlsanity.com/paper/2501.18898/) (2025)
- [DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversations](https://mlsanity.com/paper/2505.18096/) (2025)
- [Ready-to-React: Online Reaction Policy for Two-Character Interaction Generation](https://mlsanity.com/paper/2502.20370/) (2025)
- [Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis](https://mlsanity.com/paper/2412.06786/) (2024)

## Built on by (most cited first, 3 of 3)

- [Human-Centric Intelligence in the Era of Foundation Models: A Survey](https://mlsanity.com/paper/2608.18184/) (2026)
- [InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control](https://mlsanity.com/paper/2608.25734/) (2026)
- [Full-Duplex Interaction with Speech, Expressions and Gestures](https://mlsanity.com/paper/2610.21452/) (2026)

## Abstract (arXiv)

We present DyaPlex, a streaming, full-duplex speech-and-motion model designed for dyadic interaction. To capture the continuous and reciprocal nature of human communication, this full-duplex capability empowers the agent to simultaneously perceive and generate both speech and physical motion in a streaming fashion. At its core, our method leverages the strong priors of a foundational full-duplex speech model and integrates a novel motion pathway, thereby achieving fully synchronized multi-modal interaction. Specifically, we design a dual-tower Transformer architecture that preserves the zero-shot conversational reasoning of a frozen base speech model while constructing a deeply coupled, streaming motion pathway. By introducing a unified dyadic token interleaving mechanism and guiding cross-attention via a time-aligned speech-motion RoPE, our model effectively aligns autoregressive motions with rich latent speech features. Trained on the 4,000-hour Seamless Interaction dataset, our model effectively captures cross-speaker dependencies and establishes new state-of-the-art performance across both monadic and dyadic human interaction benchmarks.