mlsanityData to 14 Oct 2026

Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis

arXiv 2306.09417 ↗First posted 15 Jun 2023eess.AS cs.AI cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

With read-aloud speech synthesis achieving high naturalness scores, there is a growing research interest in synthesising spontaneous speech. However, human spontaneous face-to-face conversation has both spoken and non-verbal aspects (here, co-speech gestures). Only recently has research begun to explore the benefits of jointly synthesising these two modalities in a single system. The previous state of the art used non-probabilistic methods, which fail to capture the variability of human speech and motion, and risk producing oversmoothing artefacts and sub-optimal synthesis quality. We present the first diffusion-based probabilistic model, called Diff-TTSG, that jointly learns to synthesise speech and gestures together. Our method can be trained on small datasets from scratch. Furthermore, we describe a set of careful uni- and multi-modal subjective tests for evaluating integrated speech and gesture synthesis systems, and use them to validate our proposed approach. Please see https://shivammehta25.github.io/Diff-TTSG/ for video examples, data, and code.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 40 of its 55 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

40 of 55 references.

References · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  8. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  9. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  11. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  12. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
28 more
  1. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 447
  2. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  3. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 202
  4. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 190
  5. 2021A Survey on Neural Speech Synthesiscited by 184
  6. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 158
  7. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speechcited by 157
  8. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 152
  9. 2020Gesticulator: A framework for semantically-aware speech-driven gesture generationcited by 84
  10. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidancecited by 84
  11. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 84
  12. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  13. 2021WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesiscited by 65
  14. 2022The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generationcited by 40
  15. 2021A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020cited by 38
  16. 2020Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settingscited by 36
  17. 2022Grad-StyleSpeech: Any-speaker Adaptive Text-to-Speech Synthesis with Diffusion Modelscited by 34
  18. 2019DurIAN: Duration Informed Attention Network For Multimodal Synthesiscited by 33
  19. 2023Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022cited by 29
  20. 2023DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Modelcited by 22
  21. 2021Integrated Speech and Gesture Synthesiscited by 16
  22. 2018Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesiscited by 13
  23. 2021Generating coherent spontaneous speech and gesture from textcited by 12
  24. 2021Passing a Non-verbal Turing Test: Evaluating Gesture Animations Generated from Speechcited by 11
  25. 2021Neural HMMs are all you need (for high-quality attention-free TTS)cited by 8
  26. 2022OverFlow: Putting flows on top of neural transducers for better TTScited by 8
  27. 2022Prosody-controllable spontaneous TTS with neural HMMscited by 7
  28. 2018Speech waveform synthesis from MFCC sequences with generative adversarial networkscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 3; 2025: 2; 2026: 3.

13 papers.

Sort:

By area:

  1. 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 145
  2. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39
  3. 2023The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settingscited by 28
  4. 2023Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representationcited by 28
  5. 2023Unified speech and gesture synthesis using flow matchingcited by 6
  6. 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5
  7. 2025Gelina: Unified Speech and Gesture Synthesis via Interleaved Token Predictioncited by 2
  8. 2024Investigating the impact of 2D gesture representation on co-speech gesture generationcited by 0
  9. 20242D or not 2D: How Does the Dimensionality of Gesture Representation Affect 3D Co-Speech Gesture Generation?cited by 0
  10. 2025ImaGGen: Zero-Shot Generation of Co-Speech Semantic Gestures Grounded in Language and Image Inputcited by 0
  11. 2026Gesture2Speech: How Far Can Hand Movements Shape Expressive Speech?cited by 0
  12. 2026Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialoguecited by 0
1 more
  1. 2026DualTrack: Synchronized speech-gesture generation via symmetric coupling of pretrained priorscited by 0