mlsanityData to 7 Oct 2026

Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation

arXiv 2203.13161 ↗First posted 24 Mar 2022cs.CVcited by 106

Not on the map. Why?

Cited by 20 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generating speech-consistent body and gesture movements is a long-standing problem in virtual avatar creation. Previous studies often synthesize pose movement in a holistic manner, where poses of all joints are generated simultaneously. Such a straightforward pipeline fails to generate fine-grained co-speech gestures. One observation is that the hierarchical semantics in speech and the hierarchical structures of human gestures can be naturally described into multiple granularities and associated together. To fully utilize the rich connections between speech audio and human gestures, we propose a novel framework named Hierarchical Audio-to-Gesture (HA2G) for co-speech gesture generation. In HA2G, a Hierarchical Audio Learner extracts audio representations across semantic granularities. A Hierarchical Pose Inferer subsequently renders the entire human pose gradually in a hierarchical manner. To enhance the quality of synthesized gestures, we develop a contrastive learning strategy based on audio-text alignment for better audio representations. Extensive experiments and human evaluation demonstrate that the proposed method renders realistic co-speech gestures and outperforms previous methods in a clear margin. Project page: https://alvinliu0.github.io/projects/HA2G

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 49 of its 86 references that are in the map. First posted on arXiv 24 Mar 2022.

Builds on

49 of 86 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  4. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
  5. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  6. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,199
  7. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  8. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 913
  9. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  10. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  11. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  12. 2018The Sound of Pixelscited by 258
37 more
  1. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  2. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  3. 2019Learning Trajectory Dependencies for Human Motion Predictioncited by 245
  4. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  5. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
  6. 2019Learning Individual Styles of Conversational Gesturecited by 190
  7. 2021Audio-Driven Emotional Video Portraitscited by 180
  8. 2017HP-GAN: Probabilistic 3D human motion prediction via GANcited by 176
  9. 2020Monocular Expressive Body Regression through Body-Driven Attentioncited by 155
  10. 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
  11. 2019The Sound of Motionscited by 145
  12. 2020In defence of metric learning for speaker recognitioncited by 123
  13. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  14. 2018Convolutional Sequence to Sequence Model for Human Dynamicscited by 115
  15. 2018Human Motion Prediction via Spatio-Temporal Inpaintingcited by 114
  16. 2020Music Gesture for Visual Sound Separationcited by 112
  17. 2018Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robotscited by 110
  18. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  19. 2021Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agentscited by 106
  20. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  21. 2019Structured Prediction Helps 3D Human Motion Modellingcited by 96
  22. 2018Neural Kinematic Networks for Unsupervised Motion Retargettingcited by 85
  23. 2020Gesticulator: A framework for semantically-aware speech-driven gesture generationcited by 84
  24. 2020Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learningcited by 81
  25. 2017Learning Human Motion Models for Long-term Predictionscited by 67
  26. 2020Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approachcited by 64
  27. 2019Self-supervised Moving Vehicle Tracking with Stereo Soundcited by 63
  28. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  29. 2021Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templatescited by 57
  30. 2020Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separationcited by 55
  31. 2019Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformercited by 47
  32. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  33. 2020What comprises a good talking-head video generation?: A Survey and Benchmarkcited by 42
  34. 2020Audio-visual Recognition of Overlapped speech for the LRS2 datasetcited by 38
  35. 2020Generative Tweening: Long-term Inbetweening of 3D Human Motionscited by 23
  36. 2022Visual Sound Localization in the Wild by Cross-Modal Interference Erasingcited by 19
  37. 2020GlocalNet: Class-aware Long-term Human Motion Synthesiscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 27; 2024: 23; 2025: 33; 2026: 17.

106 papers.

Sort:

By area:

  1. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  2. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
  3. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 128
  4. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  5. 2023Human Motion Generation: A Surveycited by 95
  6. 2023HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splattingcited by 89
  7. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
  8. 2023HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusioncited by 68
  9. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  10. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 61
  11. 2022Audio-Driven Co-Speech Gesture Video Generationcited by 54
  12. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
38 more
  1. 2025GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modelingcited by 43
  2. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  3. 2023QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generationcited by 40
  4. 2023EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generationcited by 40
  5. 2023LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generationcited by 40
  6. 2022Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddingscited by 38
  7. 2024Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Modelcited by 34
  8. 2024TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolationcited by 32
  9. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  10. 2023Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022cited by 29
  11. 2024Towards Variable and Coordinated Holistic Co-Speech Motion Generationcited by 29
  12. 2022Learning to Dub Movies via Hierarchical Prosody Modelscited by 28
  13. 2025Human Motion Video Generation: A Surveycited by 25
  14. 2024A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insightscited by 24
  15. 2024MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controlscited by 24
  16. 2022Static and Dynamic Concepts for Self-supervised Video Representation Learningcited by 23
  17. 2023BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformercited by 21
  18. 2024DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gesturescited by 19
  19. 2025ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generationcited by 19
  20. 2023Diverse 3D Hand Gesture Prediction from Body Dynamics by Bilateral Hand Disentanglementcited by 18
  21. 2024Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representationscited by 18
  22. 2023Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generationcited by 17
  23. 2025EchoMask: Speech-Queried Attention-based Mask Modeling for Holistic Co-Speech Motion Generationcited by 17
  24. 2025SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learningcited by 17
  25. 2023EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Modelcited by 15
  26. 2023UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletonscited by 15
  27. 2025X-Dancer: Expressive Music to Human Dance Video Generationcited by 15
  28. 2023The DiffuseStyleGesture+ entry to the GENEA Challenge 2023cited by 14
  29. 2024Beyond Talking -- Generating Holistic 3D Human Dyadic Motion for Communicationcited by 14
  30. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  31. 2025Mitigating Error Accumulation in Co-Speech Motion Generation via Global Rotation Diffusion and Multi-Level Constraintscited by 12
  32. 2025MOSPA: Human Motion Generation Driven by Spatial Audiocited by 11
  33. 2023Audio2Gestures: Generating Diverse Gestures from Audiocited by 10
  34. 2025AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformerscited by 10
  35. 2025Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Modelscited by 10
  36. 2022The ReprGesture entry to the GENEA Challenge 2022cited by 9
  37. 2023Co-Speech Gesture Synthesis using Discrete Gesture Token Learningcited by 9
  38. 2025MimeQA: Towards Socially-Intelligent Nonverbal Foundation Modelscited by 9

50 most cited of 106 All 106 (JSON)