# T3M: Text Guided 3D Human Motion Synthesis from Speech

arXiv: https://arxiv.org/abs/2408.12885 · HTML: https://mlsanity.com/paper/2408.12885/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 23 Aug 2024.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [Generating Holistic 3D Human Motion from Speech](https://mlsanity.com/paper/2212.04420/) (2022)
- [EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation](https://mlsanity.com/paper/2303.11089/) (2023)
- [VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding](https://mlsanity.com/paper/2105.09996/) (2021)
- [Learning Speech-driven 3D Conversational Gestures from Video](https://mlsanity.com/paper/2102.06837/) (2021)
- [Animating Face using Disentangled Audio Representations](https://mlsanity.com/paper/1910.00726/) (2019)
- [Taming Diffusion Models for Music-driven Conducting Motion Generation](https://mlsanity.com/paper/2306.10065/) (2023)

## Built on by (most cited first, 6 of 6)

- [Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functions](https://mlsanity.com/paper/2507.02900/) (2025)
- [SVBench: Evaluation of Video Generation Models on Social Reasoning](https://mlsanity.com/paper/2512.21507/) (2025)
- [EasyTune: Efficient Step-Aware Fine-Tuning for Diffusion-Based Motion Generation](https://mlsanity.com/paper/2602.07967/) (2026)
- [SentiAvatar: Towards Expressive and Interactive Digital Humans](https://mlsanity.com/paper/2604.02908/) (2026)
- [InteracTalker: Prompt-Based Human-Object Interaction with Co-Speech Gesture Generation](https://mlsanity.com/paper/2512.12664/) (2025)
- [HarmoniDPO: Video-guided Audio Generation via Preference-Optimized Diffusion](https://mlsanity.com/paper/2608.11913/) (2026)

## Abstract (arXiv)

Speech-driven 3D motion synthesis seeks to create lifelike animations based on human speech, with potential uses in virtual reality, gaming, and the film production. Existing approaches reply solely on speech audio for motion generation, leading to inaccurate and inflexible synthesis results. To mitigate this problem, we introduce a novel text-guided 3D human motion synthesis method, termed textitT3M. Unlike traditional approaches, T3M allows precise control over motion synthesis via textual input, enhancing the degree of diversity and user customization. The experiment results demonstrate that T3M can greatly outperform the state-of-the-art methods in both quantitative metrics and qualitative evaluations. We have publicly released our code at hrefhttps://github.com/Gloria2tt/T3M.githttps://github.com/Gloria2tt/T3M.git