# GesGPT: Speech Gesture Synthesis With Text Parsing from ChatGPT

arXiv: https://arxiv.org/abs/2303.13013 · HTML: https://mlsanity.com/paper/2303.13013/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 50 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023)
- [Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory](https://mlsanity.com/paper/2203.13055/) (2022)
- [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020)
- [BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis](https://mlsanity.com/paper/2203.05297/) (2022)
- [Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2303.09119/) (2023)
- [Learning Speech-driven 3D Conversational Gestures from Video](https://mlsanity.com/paper/2102.06837/) (2021)
- [ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech](https://mlsanity.com/paper/2209.07556/) (2022)
- [A Comprehensive Review of Data-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2301.05339/) (2023)
- [Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templates](https://mlsanity.com/paper/2108.08020/) (2021)
- [The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generation](https://mlsanity.com/paper/2208.10441/) (2022)
- [Text2Video: Text-driven Talking-head Video Synthesis with Personalized Phoneme-Pose Dictionary](https://mlsanity.com/paper/2104.14631/) (2021)
- [Multimodal analysis of the predictability of hand-gesture properties](https://mlsanity.com/paper/2108.05762/) (2021)
- [SGToolkit: An Interactive Gesture Authoring Toolkit for Embodied Conversational Agents](https://mlsanity.com/paper/2108.04636/) (2021)
- [Sequence-to-Sequence Predictive Model: From Prosody To Communicative Gestures](https://mlsanity.com/paper/2008.07643/) (2020)
- [Deep Gesture Generation for Social Robots Using Type-Specific Libraries](https://mlsanity.com/paper/2210.06790/) (2022)

## Built on by (most cited first, 6 of 6)

- [Large language models in textual analysis for gesture selection](https://mlsanity.com/paper/2310.13705/) (2023)
- [LLM Gesticulator: Leveraging Large Language Models for Scalable and Controllable Co-Speech Gesture Synthesis](https://mlsanity.com/paper/2410.10851/) (2024)
- [Probing Language Models' Gesture Understanding for Enhanced Human-AI Interaction](https://mlsanity.com/paper/2401.17858/) (2024)
- [SARGes: Semantically Aligned Reliable Gesture Generation via Intent Chain](https://mlsanity.com/paper/2503.20202/) (2025)
- [SICAGE: Speaker-Independent Culture-Aware Gesture Generation using TED4C-L Dataset](https://mlsanity.com/paper/2606.30001/) (2026)
- [InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control](https://mlsanity.com/paper/2608.25734/) (2026)

## Abstract (arXiv)

Gesture synthesis has gained significant attention as a critical research field, aiming to produce contextually appropriate and natural gestures corresponding to speech or textual input. Although deep learning-based approaches have achieved remarkable progress, they often overlook the rich semantic information present in the text, leading to less expressive and meaningful gestures. In this letter, we propose GesGPT, a novel approach to gesture generation that leverages the semantic analysis capabilities of large language models , such as ChatGPT. By capitalizing on the strengths of LLMs for text analysis, we adopt a controlled approach to generate and integrate professional gestures and base gestures through a text parsing script, resulting in diverse and meaningful gestures. Firstly, our approach involves the development of prompt principles that transform gesture generation into an intention classification problem using ChatGPT. We also conduct further analysis on emphasis words and semantic words to aid in gesture generation. Subsequently, we construct a specialized gesture lexicon with multiple semantic annotations, decoupling the synthesis of gestures into professional gestures and base gestures. Finally, we merge the professional gestures with base gestures. Experimental results demonstrate that GesGPT effectively generates contextually appropriate and expressive gestures.