# Large language models in textual analysis for gesture selection

arXiv: https://arxiv.org/abs/2310.13705 · HTML: https://mlsanity.com/paper/2310.13705/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 82 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020)
- [Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots](https://mlsanity.com/paper/1810.12541/) (2018)
- [Gesticulator: A framework for semantically-aware speech-driven gesture generation](https://mlsanity.com/paper/2001.09326/) (2020)
- [ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech](https://mlsanity.com/paper/2209.07556/) (2022)
- [A Comprehensive Review of Data-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2301.05339/) (2023)
- [Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings](https://mlsanity.com/paper/2210.01448/) (2022)
- [To Rate or Not To Rate: Investigating Evaluation Methods for Generated Co-Speech Gestures](https://mlsanity.com/paper/2108.05709/) (2021)
- [Zero-Shot Style Transfer for Gesture Animation driven by Text and Speech using Adversarial Disentanglement of Multimodal Style Encoding](https://mlsanity.com/paper/2208.01917/) (2022)
- [GesGPT: Speech Gesture Synthesis With Text Parsing from ChatGPT](https://mlsanity.com/paper/2303.13013/) (2023)
- [A Broad-Coverage Deep Semantic Lexicon for Verbs](https://mlsanity.com/paper/2007.02670/) (2020)
- [Transformer Network for Semantically-Aware and Speech-Driven Upper-Face Generation](https://mlsanity.com/paper/2110.04527/) (2021)

## Built on by (most cited first, 6 of 6)

- [Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis](https://mlsanity.com/paper/2404.19622/) (2024)
- [Large Language Models for Virtual Human Gesture Selection](https://mlsanity.com/paper/2503.14408/) (2025)
- [LLAniMAtion: LLAMA Driven Gesture Animation](https://mlsanity.com/paper/2405.08042/) (2024)
- [Translating Signals to Languages for sEMG-Based Activity Recognition](https://mlsanity.com/paper/2605.22403/) (2026)
- [SICAGE: Speaker-Independent Culture-Aware Gesture Generation using TED4C-L Dataset](https://mlsanity.com/paper/2606.30001/) (2026)
- [LLM-Based Selection of Incongruent Verbal and Nonverbal Behavior for Virtual Humans](https://mlsanity.com/paper/2608.22731/) (2026)

## Abstract (arXiv)

Gestures perform a variety of communicative functions that powerfully influence human face-to-face interaction. How this communicative function is achieved varies greatly between individuals and depends on the role of the speaker and the context of the interaction. Approaches to automatic gesture generation vary not only in the degree to which they rely on data-driven techniques but also the degree to which they can produce context and speaker specific gestures. However, these approaches face two major challenges: The first is obtaining sufficient training data that is appropriate for the context and the goal of the application. The second is related to designer control to realize their specific intent for the application. Here, we approach these challenges by using large language models (LLMs) to show that these powerful models of large amounts of data can be adapted for gesture analysis and generation. Specifically, we used ChatGPT as a tool for suggesting context-specific gestures that can realize designer intent based on minimal prompts. We also find that ChatGPT can suggests novel yet appropriate gestures not present in the minimal training data. The use of LLMs is a promising avenue for gesture generation that reduce the need for laborious annotations and has the potential to flexibly and quickly adapt to different designer intents.