mlsanityData to 14 Oct 2026

LLM Gesticulator: Leveraging Large Language Models for Scalable and Controllable Co-Speech Gesture Synthesis

arXiv 2410.10851 ↗First posted 6 Oct 2024cs.GR cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

In this work, we present LLM Gesticulator, an LLM-based audio-driven co-speech gesture generation framework that synthesizes full-body animations that are rhythmically aligned with the input audio while exhibiting natural movements and editability. Compared to previous work, our model demonstrates substantial scalability. As the size of the backbone LLM model increases, our framework shows proportional improvements in evaluation metrics (a.k.a. scaling law). Our method also exhibits strong controllability where the content, style of the generated gestures can be controlled by text prompt. To the best of our knowledge, LLM gesticulator is the first work that use LLM on the co-speech generation task. Evaluation with existing objective metrics and user studies indicate that our framework outperforms prior works.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 6 Oct 2024.

Builds on

30 of 46 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  4. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  5. 2022High Fidelity Neural Audio Compressioncited by 687
  6. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 575
  7. 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
  8. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 485
  9. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  10. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 202
  11. 2019Learning Individual Styles of Conversational Gesturecited by 191
  12. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 178
18 more
  1. 2018Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robotscited by 110
  2. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  3. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 107
  4. 2021Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agentscited by 106
  5. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  6. 2020Gesticulator: A framework for semantically-aware speech-driven gesture generationcited by 84
  7. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 84
  8. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  9. 2023MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representationscited by 60
  10. 2021Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learningcited by 58
  11. 2021Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templatescited by 58
  12. 2019Analyzing Input and Output Representations for Speech-Driven Gesture Generationcited by 52
  13. 2019Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interactioncited by 48
  14. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  15. 2024Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesiscited by 40
  16. 2023Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022cited by 29
  17. 2022Freeform Body Motion Generation from Speechcited by 11
  18. 2023GesGPT: Speech Gesture Synthesis With Text Parsing from ChatGPTcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  2. 2025Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Modelscited by 11
  3. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  4. 2026Botson: An Accessible and Low-Cost Platform for Social Robotics Researchcited by 0
  5. 2026LLM-Based Selection of Incongruent Verbal and Nonverbal Behavior for Virtual Humanscited by 0