# Large Language Models for Virtual Human Gesture Selection

arXiv: https://arxiv.org/abs/2503.14408 · HTML: https://mlsanity.com/paper/2503.14408/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 49 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Builds on (most cited first, 4 of 4)

- [A Comprehensive Review of Data-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2301.05339/) (2023)
- [DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model](https://mlsanity.com/paper/2301.10047/) (2023)
- [Zero-Shot Style Transfer for Gesture Animation driven by Text and Speech using Adversarial Disentanglement of Multimodal Style Encoding](https://mlsanity.com/paper/2208.01917/) (2022)
- [Large language models in textual analysis for gesture selection](https://mlsanity.com/paper/2310.13705/) (2023)

## Built on by (most cited first, 4 of 4)

- [Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agents](https://mlsanity.com/paper/2510.04637/) (2025)
- [Can LLMs Generate Behaviors for Embodied Virtual Agents Based on Personality Traits?](https://mlsanity.com/paper/2508.21087/) (2025)
- [Translating Signals to Languages for sEMG-Based Activity Recognition](https://mlsanity.com/paper/2605.22403/) (2026)
- [LLM-Based Selection of Incongruent Verbal and Nonverbal Behavior for Virtual Humans](https://mlsanity.com/paper/2608.22731/) (2026)

## Abstract (arXiv)

Co-speech gestures convey a wide variety of meanings and play an important role in face-to-face human interactions. These gestures significantly influence the addressee's engagement, recall, comprehension, and attitudes toward the speaker. Similarly, they impact interactions between humans and embodied virtual agents. The process of selecting and animating meaningful gestures has thus become a key focus in the design of these agents. However, automating this gesture selection process poses a significant challenge. Prior gesture generation techniques have varied from fully automated, data-driven methods, which often struggle to produce contextually meaningful gestures, to more manual approaches that require crafting specific gesture expertise and are time-consuming and lack generalizability. In this paper, we leverage the semantic capabilities of Large Language Models to develop a gesture selection approach that suggests meaningful, appropriate co-speech gestures. We first describe how information on gestures is encoded into GPT-4. Then, we conduct a study to evaluate alternative prompting approaches for their ability to select meaningful, contextually relevant gestures and to align them appropriately with the co-speech utterance. Finally, we detail and demonstrate how this approach has been implemented within a virtual agent system, automating the selection and subsequent animation of the selected gestures for enhanced human-agent interactions.