mlsanityData to 7 Oct 2026

LLAniMAtion: LLAMA Driven Gesture Animation

arXiv 2405.08042 ↗First posted 13 May 2024cs.HC cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Co-speech gesturing is an important modality in conversation, providing context and social cues. In character animation, appropriate and synchronised gestures add realism, and can make interactive agents more engaging. Historically, methods for automatically generating gestures were predominantly audio-driven, exploiting the prosodic and speech-related content that is encoded in the audio signal. In this paper we instead experiment with using LLM features for gesture generation that are extracted from text using LLAMA2. We compare against audio features, and explore combining the two modalities in both objective tests and a user study. Surprisingly, our results show that LLAMA2 features on their own perform significantly better than audio features and that including both modalities yields no significant difference to using LLAMA2 features in isolation. We demonstrate that the LLAMA2 based model can generate both beat and semantic gestures without any audio input, suggesting LLMs can provide rich encodings that are well suited for gesture generation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 46 references that are in the map. First posted on arXiv 13 May 2024.

Builds on

23 of 46 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  6. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  7. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  8. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  9. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
  10. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  11. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  12. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
11 more
  1. 2021Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agentscited by 106
  2. 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82
  3. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  4. 2021Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learningcited by 58
  5. 2021Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templatescited by 57
  6. 2019Analyzing Input and Output Representations for Speech-Driven Gesture Generationcited by 51
  7. 2022Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddingscited by 38
  8. 2023The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settingscited by 27
  9. 2023Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representationcited by 27
  10. 2023DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Modelcited by 22
  11. 2023Large language models in textual analysis for gesture selectioncited by 6

Built on by

Not cited yet.