GDPO-Listener: Expressive Interactive Head Generation via Auto-Regressive Flow Matching and Group reward-Decoupled Policy Optimization
On the map: Diffusion models → Audio, speech and music
Abstract (arXiv)
Generating realistic 3D head motion for dyadic interactions is a significant challenge in virtual human synthesis. While recent methods achieve impressive results with speaking heads, they frequently suffer from the `Regression-to-the-Mean' problem in listener motions, collapsing into static faces, and lack the parameter space for complex nonverbal motions. In this paper, we propose GDPO-Listener, a novel framework that achieves highly expressive speaking and listening motion generation. First, we introduce an Auto-Regressive Flow Matching architecture enabling stable supervised learning. Second, to overcome kinematic stillness, we apply the Group reward-Decoupled Policy Optimization (GDPO). By isolating reward normalization across distinct FLAME parameter groups, GDPO explicitly incentivizes high variance expressive generations. Finally, we enable explicit semantic text control for customizable responses. Extensive evaluations across the Seamless Interaction and DualTalk datasets demonstrate superior performance compared to existing baselines on long-term kinematic variance, visual expressivity and semantic controllability.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 33 of its 39 references that are in the map. First posted on arXiv 26 Mar 2026.
Builds on
33 of 39 references.
Closest ancestors · 3
Same direction, newest first
- 2025DiTaiListener: Controllable High Fidelity Listener Video Generation with Diffusion5 Apr
- 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models30 Sep
- 2022Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors7 Dec
Other references · 30
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2022Classifier-Free Diffusion Guidancecited by 5,847
- 2017Neural Discrete Representation Learningcited by 4,712
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
- 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
- 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 209
- 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 193
- 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 190
18 more
- 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 184
- 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 126
- 2024Image and Video Tokenization with Binary Spherical Quantizationcited by 112
- 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 108
- 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82
- 2021Responsive Listening Head Generation: A Benchmark Dataset and Baselinecited by 67
- 2023Can Language Models Learn to Listen?cited by 40
- 2024INFP: Audio-Driven Interactive Head Generation in Dyadic Conversationscited by 40
- 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 34
- 2024Dyadic Interaction Modeling for Social Behavior Generationcited by 32
- 2025DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversationscited by 32
- 2025Taming Teacher Forcing for Masked Autoregressive Video Generationcited by 30
- 2025ARTalk: Speech-Driven 3D Head Animation via Autoregressive Modelcited by 29
- 2024CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generationcited by 22
- 2026Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversationcited by 19
- 2025UniLS: End-to-End Audio-Driven Avatars for Unified Listening and Speakingcited by 11
- 2025SHeaP: Self-Supervised Head Geometry Predictor Learned via 2D Gaussianscited by 8
- 2026FlowPortrait: Reinforcement Learning for Audio-Driven Portrait Video Generationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026MindFlow: Harmonizing Cognitive Semantics and Acoustic Dynamics for Facial Animation Generation in Dyadic Conversations
- 2026STEER: Steerable Dyadic Head Avatars
- 2026EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents
- 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models
- 2026InterDyad: Interactive Dyadic Speech-to-Video Generation by Querying Intermediate Visual Guidance
- 2024Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis
- 2025ARIG: Autoregressive Interactive Head Generation for Real-time Conversations
- 2025DEMO: Disentangled Motion Latent Flow Matching for Fine-Grained Controllable Talking Portrait Synthesis