mlsanityData to 14 Oct 2026

Unified speech and gesture synthesis using flow matching

arXiv 2310.05181 ↗First posted 8 Oct 2023eess.AS cs.GR cs.HCcited by 6

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

As text-to-speech technologies achieve remarkable naturalness in read-aloud tasks, there is growing interest in multimodal synthesis of verbal and non-verbal communicative behaviour, such as spontaneous speech and associated body gestures. This paper presents a novel, unified architecture for jointly synthesising speech acoustics and skeleton-based 3D gesture motion from text, trained using optimal-transport conditional flow matching (OT-CFM). The proposed architecture is simpler than the previous state of the art, has a smaller memory footprint, and can capture the joint distribution of speech and gestures, generating both modalities together in one single process. The new training regime, meanwhile, enables better synthesis quality in much fewer steps (network evaluations) than before. Uni- and multimodal subjective tests demonstrate improved speech naturalness, gesture human-likeness, and cross-modal appropriateness compared to existing benchmarks. Please see https://shivammehta25.github.io/Match-TTSG/ for video examples and code.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 32 of its 43 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

32 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents26 Mar
  2. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models17 Nov
  3. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May

Other references · 29

  1. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  2. 2022Flow Matching for Generative Modelingcited by 5,702
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2018Neural Ordinary Differential Equationscited by 3,779
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  7. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  8. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  9. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  10. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  11. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  12. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309
17 more
  1. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  2. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  3. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  4. 2022The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generationcited by 40
  5. 2021A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020cited by 38
  6. 2020Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settingscited by 36
  7. 2019DurIAN: Duration Informed Attention Network For Multimodal Synthesiscited by 33
  8. 2023Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022cited by 29
  9. 2023The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settingscited by 28
  10. 2023Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representationcited by 28
  11. 2021The Case for Translation-Invariant Self-Attention in Transformer-Based Language Modelscited by 23
  12. 2021Integrated Speech and Gesture Synthesiscited by 16
  13. 2023Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesiscited by 13
  14. 2021Generating coherent spontaneous speech and gesture from textcited by 12
  15. 2022OverFlow: Putting flows on top of neural transducers for better TTScited by 8
  16. 2023UniFLG: Unified Facial Landmark Generator from Text or Speechcited by 8
  17. 2023On the Use of Self-Supervised Speech Representations in Spontaneous Speech Synthesiscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 3.

6 papers.

  1. 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5
  2. 2025Gelina: Unified Speech and Gesture Synthesis via Interleaved Token Predictioncited by 2
  3. 2025ImaGGen: Zero-Shot Generation of Co-Speech Semantic Gestures Grounded in Language and Image Inputcited by 0
  4. 2026Gesture2Speech: How Far Can Hand Movements Shape Expressive Speech?cited by 0
  5. 2026Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialoguecited by 0
  6. 2026DualTrack: Synchronized speech-gesture generation via symmetric coupling of pretrained priorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SemConFlow: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matching
  2. 2025GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling
  3. 2025GestureHYDRA: Semantic Co-speech Gesture Synthesis via Hybrid Modality Diffusion Transformer and Cascaded-Synchronized Retrieval-Augmented Generation
  4. 2023DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model
  5. 2023QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation
  6. 2023EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model
  7. 2024Faces that Speak: Jointly Synthesising Talking Face and Speech from Text
  8. 2023UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons