mlsanityData to 7 Oct 2026

EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling

arXiv 2401.00374 ↗First posted 31 Dec 2023cs.CVcited by 101

Not on the map. Why?

Abstract (arXiv)

We propose EMAGE, a framework to generate full-body human gestures from audio and masked gestures, encompassing facial, local body, hands, and global movements. To achieve this, we first introduce BEAT2 (BEAT-SMPLX-FLAME), a new mesh-level holistic co-speech dataset. BEAT2 combines a MoShed SMPL-X body with FLAME head parameters and further refines the modeling of head, neck, and finger movements, offering a community-standardized, high-quality 3D motion captured dataset. EMAGE leverages masked body gesture priors during training to boost inference performance. It involves a Masked Audio Gesture Transformer, facilitating joint training on audio-to-gesture generation and masked gesture reconstruction to effectively encode audio and body gesture hints. Encoded body hints from masked gestures are then separately employed to generate facial and body movements. Moreover, EMAGE adaptively merges speech features from the audio's rhythm and content and utilizes four compositional VQ-VAEs to enhance the results' fidelity and diversity. Experiments demonstrate that EMAGE generates holistic gestures with state-of-the-art performance and is flexible in accepting predefined spatial-temporal gesture inputs, generating complete, audio-synchronized results. Our code and dataset are available https://pantomatrix.github.io/EMAGE/

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 58 of its 81 references that are in the map. First posted on arXiv 31 Dec 2023.

Builds on

58 of 81 references.

References · 58

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  10. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  11. 2020Point Transformercited by 1,462
  12. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
38 more
  1. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  2. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  3. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  4. 2021AST: Audio Spectrogram Transformercited by 634
  5. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  6. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  7. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  8. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 357
  9. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
  10. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  11. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
  12. 2019Learning Individual Styles of Conversational Gesturecited by 190
  13. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  14. 2022MotionBERT: A Unified Perspective on Learning Human Motion Representationscited by 188
  15. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  16. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  17. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  18. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
  19. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 141
  20. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 128
  21. 2020Skeleton-Aware Networks for Deep Motion Retargetingcited by 118
  22. 2018Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robotscited by 110
  23. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  24. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
  25. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  26. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  27. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
  28. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 83
  29. 2023BlendFace: Re-designing Identity Encoders for Face-Swappingcited by 66
  30. 2020Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approachcited by 64
  31. 2021Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templatescited by 57
  32. 2021Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generationcited by 46
  33. 2023QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generationcited by 40
  34. 2023EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generationcited by 40
  35. 2023LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generationcited by 40
  36. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39
  37. 2022Differentiable Dynamics for Articulated 3d Human Motion Reconstructioncited by 38
  38. 2020Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamicscited by 34

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 41; 2026: 48.

101 papers.

Sort:

By area:

  1. 2025Controllable Video Generation: A Surveycited by 93
  2. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  3. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  4. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  5. 2024Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesiscited by 25
  6. 2024MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controlscited by 24
  7. 2025BEDLAM2.0: Synthetic Humans and Cameras in Motioncited by 21
  8. 2025The Quest for Generalizable Motion Generation: Data, Model, and Evaluationcited by 20
  9. 2024DisCoRD: Discrete Tokens to Continuous Motion via Rectified Flow Decodingcited by 17
  10. 2025EchoMask: Speech-Queried Attention-based Mask Modeling for Holistic Co-Speech Motion Generationcited by 17
  11. 2025SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learningcited by 17
  12. 2025SocialGen: Modeling Multi-Human Social Interaction with Language Modelscited by 14
38 more
  1. 2024ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAEcited by 13
  2. 2025DPoser-X: Diffusion Model as Robust 3D Whole-body Human Pose Priorcited by 13
  3. 2024DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animationcited by 12
  4. 2025Mitigating Error Accumulation in Co-Speech Motion Generation via Global Rotation Diffusion and Multi-Level Constraintscited by 12
  5. 2025Co3Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusioncited by 9
  6. 2025MemoryTalker: Personalized Speech-Driven 3D Facial Animation via Audio-Guided Stylizationcited by 9
  7. 2026MIBURI: Towards Expressive Interactive Gesture Synthesiscited by 9
  8. 2025Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animationcited by 8
  9. 2026DyaDiT: A Multi-Modal Diffusion Transformer for Socially Favorable Dyadic Gesture Generationcited by 7
  10. 2026PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakerscited by 7
  11. 2024Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmonycited by 6
  12. 2025HOP: Heterogeneous Topology-based Multimodal Entanglement for Co-Speech Gesture Generationcited by 6
  13. 2025GestureHYDRA: Semantic Co-speech Gesture Synthesis via Hybrid Modality Diffusion Transformer and Cascaded-Synchronized Retrieval-Augmented Generationcited by 6
  14. 2025ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Bodycited by 6
  15. 2026DuoMo: Dual Motion Diffusion for World-Space Human Reconstructioncited by 6
  16. 2025GenM3: Generative Pretrained Multi-path Motion Model for Text Conditional Human Motion Generationcited by 5
  17. 2025Video Motion Graphscited by 5
  18. 2026Temporal Consistency-Aware Text-to-Motion Generationcited by 5
  19. 2025Speaking Beyond Language: A Large-Scale Multimodal Dataset for Learning Nonverbal Cues from Video-Grounded Dialoguescited by 4
  20. 2025CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generationcited by 4
  21. 2026SemConFlow: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matchingcited by 4
  22. 2026Efficient Emotion-Aware Iconic Gesture Prediction for Robot Co-Speechcited by 4
  23. 2026SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-Based Humanoid Controlcited by 4
  24. 2024Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesiscited by 3
  25. 2025Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformerscited by 3
  26. 2025PA-HOI: A Physics-Aware Human and Object Interaction Datasetcited by 3
  27. 2025OmniMotion: Multimodal Motion Generation with Continuous Masked Autoregressioncited by 3
  28. 2025fMRI2GES: Co-speech Gesture Reconstruction from fMRI Signal with Dual Brain Decoding Alignmentcited by 3
  29. 2025InteracTalker: Prompt-Based Human-Object Interaction with Co-Speech Gesture Generationcited by 3
  30. 2026SentiAvatar: Towards Expressive and Interactive Digital Humanscited by 3
  31. 2026PhysDrift: Bridging the Embodiment Gap in Humanoid Co-Speech Motion Generationcited by 3
  32. 2024Diverse Code Query Learning for Speech-Driven Facial Animationcited by 2
  33. 2025Enhancing Spoken Discourse Modeling in Language Models Using Gestural Cuescited by 2
  34. 2025Gelina: Unified Speech and Gesture Synthesis via Interleaved Token Predictioncited by 2
  35. 2025MimicParts: Part-aware Style Injection for Speech-Driven 3D Motion Generationcited by 2
  36. 2026ProAct: Harnessing Streaming Motion Generation and Agentic Reasoning for Real-Time Embodied Social Interactioncited by 2
  37. 2026SARAH: Spatially Aware Real-time Agentic Humanscited by 2
  38. 2026U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generationcited by 2

50 most cited of 101 All 101 (JSON)