mlsanityData to 14 Oct 2026

Contextual Gesture: Co-Speech Gesture Video Generation through Context-aware Gesture Representation

arXiv 2502.07239 ↗First posted 11 Feb 2025cs.CV cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Co-speech gesture generation is crucial for creating lifelike avatars and enhancing human-computer interactions by synchronizing gestures with speech. Despite recent advancements, existing methods struggle with accurately identifying the rhythmic or semantic triggers from audio for generating contextualized gesture patterns and achieving pixel-level realism. To address these challenges, we introduce Contextual Gesture, a framework that improves co-speech gesture video generation through three innovative components: (1) a chronological speech-gesture alignment that temporally connects two modalities, (2) a contextualized gesture tokenization that incorporate speech context into motion pattern representation through distillation, and (3) a structure-aware refinement module that employs edge connection to link gesture keypoints to improve video generation. Our extensive experiments demonstrate that Contextual Gesture not only produces realistic and speech-aligned gesture videos but also supports long-sequence generation and video gesture editing applications, shown in Fig.1.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 57 of its 90 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

57 of 90 references.

References · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  7. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  9. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  10. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  11. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  12. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
38 more
  1. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  2. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  3. 2020First Order Motion Model for Image Animationcited by 704
  4. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 688
  5. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 485
  6. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  7. 2022Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectivescited by 320
  8. 2018Everybody Dance Nowcited by 315
  9. 2021Motion Representations for Articulated Animationcited by 255
  10. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227
  11. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 227
  12. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 212
  13. 2022Thin-Plate Spline Motion Model for Image Animationcited by 210
  14. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 202
  15. 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 198
  16. 2019Learning Individual Styles of Conversational Gesturecited by 191
  17. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 184
  18. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  19. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  20. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 178
  21. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
  22. 2023MMM: Generative Masked Motion Modelcited by 131
  23. 2024EraseAnything: Enabling Concept Erasure in Rectified Flow Transformerscited by 95
  24. 2024EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animationcited by 72
  25. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 62
  26. 2022Audio-Driven Co-Speech Gesture Video Generationcited by 55
  27. 2025GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modelingcited by 44
  28. 2022Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddingscited by 38
  29. 2024DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Spacecited by 34
  30. 2024TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolationcited by 32
  31. 2023Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representationcited by 28
  32. 2024OFTSR: One-Step Flow for Image Super-Resolution with Tunable Fidelity-Realism Trade-offscited by 22
  33. 2024TextToon: Real-Time Text Toonify Head Avatar from Single Videocited by 21
  34. 2023Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Controlcited by 18
  35. 2022AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypointscited by 17
  36. 2023Emotional Listener Portrait: Neural Listener Head Generation with Emotioncited by 17
  37. 2024Adaptive Super Resolution For One-Shot Talking-Head Generationcited by 15
  38. 2024CaRDiff: Video Salient Object Ranking Chain of Thought Reasoning for Saliency Prediction with Diffusioncited by 12

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 5.

12 papers.

  1. 2025Intentional Gesture: Deliver Your Intentions with Gestures for Speechcited by 8
  2. 2025Video-R4: Reinforcing Text-Rich Video Reasoning with Visual Ruminationcited by 6
  3. 2026LiveGesture Streamable Co-Speech Gesture Generation Modelcited by 5
  4. 2026SemConFlow: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matchingcited by 4
  5. 2025MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animationcited by 3
  6. 2025Diffusion-Based Image Editing: An Unforeseen Adversary to Robust Invisible Watermarkscited by 3
  7. 2026GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregressioncited by 1
  8. 2025Semantics-Aware Human Motion Generation from Audio Instructionscited by 0
  9. 2025FADE: Adversarial Concept Erasure in Flow Modelscited by 0
  10. 2025Robust Concept Erasure in Diffusion Models: A Theoretical Perspective on Security and Robustnesscited by 0
  11. 2026TDMM-LM: Bridging Facial Understanding and Animation via Language Modelscited by 0
  12. 2026InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Controlcited by 0