mlsanityData to 7 Oct 2026

Learning Individual Styles of Conversational Gesture

arXiv 1906.04160 ↗First posted 10 Jun 2019cs.CV cs.LG eess.AScited by 190

Not on the map. Why?

Cited by 26 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Human speech is often accompanied by hand and arm gestures. Given audio speech input, we generate plausible gestures to go along with the sound. Specifically, we perform cross-modal translation from "in-the-wild'' monologue speech of a single speaker to their hand and arm motion. We train on unlabeled videos for which we only have noisy pseudo ground truth from an automatic pose detection system. Our proposed model significantly outperforms baseline methods in a quantitative comparison. To support research toward obtaining a computational understanding of the relationship between gesture and speech, we release a large video dataset of person-specific gestures. The project website with video, code and data can be found at http://people.eecs.berkeley.edu/~shiry/speech2gesture .

Key facts. Cited by 190 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 22; 2023: 42; 2024: 42; 2025: 40; 2026: 27.

190 papers.

Sort:

By area:

  1. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  2. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  3. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  4. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  5. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 210
  6. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  7. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  8. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
  9. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 128
  10. 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 125
  11. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  12. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
38 more
  1. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  2. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  3. 2023Human Motion Generation: A Surveycited by 95
  4. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
  5. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 83
  6. 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82
  7. 2021Driving-Signal Aware Full-Body Avatarscited by 81
  8. 2023TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integrationcited by 76
  9. 2021Responsive Listening Head Generation: A Benchmark Dataset and Baselinecited by 67
  10. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  11. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  12. 2021Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learningcited by 58
  13. 2021Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templatescited by 57
  14. 2022Audio-Driven Co-Speech Gesture Video Generationcited by 54
  15. 2021Human Hands as Probes for Interactive Object Understandingcited by 51
  16. 2021Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networkscited by 50
  17. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  18. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  19. 2025GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modelingcited by 43
  20. 2023Sound to Visual Scene Generation by Audio-to-Visual Latent Alignmentcited by 41
  21. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  22. 2023QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generationcited by 40
  23. 2023EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generationcited by 40
  24. 2023Can Language Models Learn to Listen?cited by 40
  25. 2023LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generationcited by 40
  26. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39
  27. 2022Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddingscited by 38
  28. 2024CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attentioncited by 38
  29. 2025HumanDiT: Pose-Guided Diffusion Transformer for Long-form Human Motion Video Generationcited by 38
  30. 2024Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generationcited by 35
  31. 2024Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Modelcited by 34
  32. 2024TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolationcited by 32
  33. 2022The One Where They Reconstructed 3D Humans and Environments in TV Showscited by 30
  34. 2024Towards Variable and Coordinated Holistic Co-Speech Motion Generationcited by 29
  35. 2023Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answeringcited by 28
  36. 2021Text2Video: Text-driven Talking-head Video Synthesis with Personalized Phoneme-Pose Dictionarycited by 27
  37. 2024Make-Your-Anchor: A Diffusion-based 2D Avatar Generation Frameworkcited by 26
  38. 2024A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insightscited by 24

50 most cited of 190 All 190 (JSON)