mlsanityData to 14 Oct 2026

Joint Co-Speech Gesture and Expressive Talking Face Generation using Diffusion with Adapters

arXiv 2412.14333 ↗First posted 18 Dec 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent advances in co-speech gesture and talking head generation have been impressive, yet most methods focus on only one of the two tasks. Those that attempt to generate both often rely on separate models or network modules, increasing training complexity and ignoring the inherent relationship between face and body movements. To address the challenges, in this paper, we propose a novel model architecture that jointly generates face and body motions within a single network. This approach leverages shared weights between modalities, facilitated by adapters that enable adaptation to a common latent space. Our experiments demonstrate that the proposed framework not only maintains state-of-the-art co-speech gesture and talking head generation performance but also significantly reduces the number of parameters required.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 45 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

37 of 45 references.

References · 37

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  8. 2022Human Motion Diffusion Modelcited by 1,080
  9. 2017Learning multiple visual domains with residual adapterscited by 557
  10. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  11. 2022EDGE: Editable Dance Generation From Musiccited by 287
  12. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
25 more
  1. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulecited by 211
  2. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 209
  3. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 202
  4. 2019Learning Individual Styles of Conversational Gesturecited by 191
  5. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 184
  6. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  7. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 178
  8. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 158
  9. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  10. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animationcited by 136
  11. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 129
  12. 2018Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robotscited by 110
  13. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  14. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 107
  15. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 105
  16. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  17. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
  18. 2022Imitator: Personalized Speech-driven 3D Facial Animationcited by 74
  19. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  20. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 62
  21. 2021Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templatescited by 58
  22. 2022Audio-Driven Co-Speech Gesture Video Generationcited by 55
  23. 2022Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddingscited by 38
  24. 2023Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representationcited by 28
  25. 2023BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformercited by 21

Built on by

Not cited yet.