mlsanityData to 7 Oct 2026

DualTrack: Synchronized speech-gesture generation via symmetric coupling of pretrained priors

arXiv 2609.36624 ↗First posted 29 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Joint speech-gesture synthesis must coordinate two modalities despite limited paired data. Existing approaches often lack bidirectional interaction, have limited language coverage, or simplify body and finger representations. We present DualTrack, which couples pretrained speech and motion priors on a shared 12.5 Hz timeline. Causal adapters exchange previous-packet information, while current-state fusion coordinates the streams before they separately complete sixteen-codebook packets. We evaluate 43 BEAT2 recordings in four languages, with speakers held out from joint training and validation. On the shared English/Spanish inputs, without speech or motion prefixes, DualTrack achieves lower word error rate and full-motion Fr'echet Gesture Distance, higher beat consistency and speech naturalness than the evaluated GELINA baseline.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 26 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

22 of 26 references.

References · 22

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  3. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  4. 2022High Fidelity Neural Audio Compressioncited by 673
  5. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  6. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  7. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  8. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  9. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
  10. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 189
  11. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  12. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 128
10 more
  1. 2026Qwen3-TTS Technical Reportcited by 106
  2. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  3. 2020Gesticulator: A framework for semantically-aware speech-driven gesture generationcited by 84
  4. 2021Integrated Speech and Gesture Synthesiscited by 16
  5. 2023Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesiscited by 13
  6. 2023Unified speech and gesture synthesis using flow matchingcited by 6
  7. 2026SOMA: Unifying Parametric Human Body Modelscited by 6
  8. 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5
  9. 2025Gelina: Unified Speech and Gesture Synthesis via Interleaved Token Predictioncited by 2
  10. 2026SeMoCo: A Semantic-First Motion Codec for Motion Language Modelingcited by 1

Built on by

Not cited yet.