Gelina: Unified Speech and Gesture Synthesis via Interleaved Token Prediction
Not on the map. Why?
Abstract (arXiv)
Human communication is multimodal, with speech and gestures tightly coupled, yet most computational methods for generating speech and gestures synthesize them sequentially, weakening synchrony and prosody alignment. We introduce Gelina, a unified framework that jointly synthesizes speech and co-speech gestures from text using interleaved token sequences in a discrete autoregressive backbone, with modality-specific decoders. Gelina supports multi-speaker and multi-style cloning and enables gesture-only synthesis from speech inputs. Subjective and objective evaluations demonstrate competitive speech quality and improved gesture generation over unimodal baselines.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 31 of its 37 references that are in the map. First posted on arXiv 13 Oct 2025.
Builds on
31 of 37 references.
References · 31
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
- 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
- 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
- 2022High Fidelity Neural Audio Compressioncited by 673
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
- 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
- 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
- 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
19 more
- 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
- 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
- 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
- 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 189
- 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
- 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
- 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 142
- 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
- 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
- 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
- 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
- 2023Better speech synthesis through scalingcited by 63
- 2024Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesiscited by 25
- 2021Integrated Speech and Gesture Synthesiscited by 16
- 2023Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesiscited by 13
- 2023Unified speech and gesture synthesis using flow matchingcited by 6
- 2024Small-E: Small Language Model with Linear Attention for Efficient Speech Synthesiscited by 6
- 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5
- 2024FastTalker: Jointly Generating Speech and Conversational Gestures from Textcited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.