# LiveGesture Streamable Co-Speech Gesture Generation Model

arXiv: https://arxiv.org/abs/2604.10927 · HTML: https://mlsanity.com/paper/2604.10927/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 49 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Builds on (most cited first, 30 of 42)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020)
- [Learning Individual Styles of Conversational Gesture](https://mlsanity.com/paper/1906.04160/) (2019)
- [CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior](https://mlsanity.com/paper/2301.02379/) (2023)
- [Generating Holistic 3D Human Motion from Speech](https://mlsanity.com/paper/2212.04420/) (2022)
- [BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis](https://mlsanity.com/paper/2203.05297/) (2022)
- [MMM: Generative Masked Motion Model](https://mlsanity.com/paper/2312.03596/) (2023)
- [AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanism](https://mlsanity.com/paper/2309.00796/) (2023)
- [Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders](https://mlsanity.com/paper/2108.06720/) (2021)
- [Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation](https://mlsanity.com/paper/2203.13161/) (2022)
- [Learning Speech-driven 3D Conversational Gestures from Video](https://mlsanity.com/paper/2102.06837/) (2021)
- [DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation](https://mlsanity.com/paper/2401.04747/) (2024)
- [BAMM: Bidirectional Autoregressive Motion Model](https://mlsanity.com/paper/2403.19435/) (2024)
- [MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models](https://mlsanity.com/paper/2403.09471/) (2024)
- [GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling](https://mlsanity.com/paper/2501.18898/) (2025)
- [Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generation](https://mlsanity.com/paper/2410.00464/) (2024)
- [TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolation](https://mlsanity.com/paper/2410.04221/) (2024)
- [VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model](https://mlsanity.com/paper/2505.03739/) (2025)
- [Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representation](https://mlsanity.com/paper/2309.05455/) (2023)

## Built on by (most cited first, 5 of 5)

- [PhysDrift: Bridging the Embodiment Gap in Humanoid Co-Speech Motion Generation](https://mlsanity.com/paper/2606.19935/) (2026)
- [InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control](https://mlsanity.com/paper/2608.25734/) (2026)
- [SocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generation](https://mlsanity.com/paper/2609.33311/) (2026)
- [When Semantics Matter: Reliability-Aware Semantic-Rhythm Control for Co-Speech Gesture Generation](https://mlsanity.com/paper/2609.36685/) (2026)
- [Sonic-AMG: Streaming Human Motion Generation from Spatial Audio](https://mlsanity.com/paper/2610.13300/) (2026)

## Abstract (arXiv)

We propose LiveGesture, the first fully streamable, speech-driven full-body gesture generation framework that operates with zero look-ahead and supports arbitrary sequence length. Unlike existing co-speech gesture methods, which are designed for offline generation and either treat body regions independently or entangle all joints within a single model, LiveGesture is built from the ground up for causal, region-coordinated motion generation. LiveGesture consists of two main modules: the Streamable Vector Quantized Motion Tokenizer (SVQ) and the Hierarchical Autoregressive Transformer (HAR). The SVQ tokenizer converts the motion sequence of each body region into causal, discrete motion tokens, enabling real-time, streamable token decoding. On top of SVQ, HAR employs region-expert autoregressive (xAR) transformers to model expressive, fine-grained motion dynamics for each body region. A causal spatio-temporal fusion module (xAR Fusion) then captures and integrates correlated motion dynamics across regions. Both xAR and xAR Fusion are conditioned on live, continuously arriving audio signals encoded by a streamable causal audio encoder. To enhance robustness under streaming noise and prediction errors, we introduce autoregressive masking training, which leverages uncertainty-guided token masking and random region masking to expose the model to imperfect, partially erroneous histories during training. Experiments on the BEAT2 dataset demonstrate that LiveGesture produces coherent, diverse, and beat-synchronous full-body gestures in real time, matching or surpassing state-of-the-art offline methods under true zero look-ahead conditions.