mlsanityData to 14 Oct 2026

JWB-DH-V1: Benchmark for Joint Whole-Body Talking Avatar and Speech Generation Version 1

arXiv 2507.20987 ↗First posted 28 Jul 2025cs.CV cs.AIcited by 0

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Recent advances in diffusion-based video generation have enabled photo-realistic short clips, but current methods still struggle to achieve multi-modal consistency when jointly generating whole-body motion and natural speech. Current approaches lack comprehensive evaluation frameworks that assess both visual and audio quality, and there are insufficient benchmarks for region-specific performance analysis. To address these gaps, we introduce the Joint Whole-Body Talking Avatar and Speech Generation Version I(JWB-DH-V1), comprising a large-scale multi-modal dataset with 10,000 unique identities across 2 million video samples, and an evaluation protocol for assessing joint audio-video generation of whole-body animatable avatars. Our evaluation of SOTA models reveals consistent performance disparities between face/hand-centric and whole-body performance, which incidates essential areas for future research. The dataset and evaluation tools are publicly available at https://github.com/deepreasonings/WholeBodyBenchmark.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 31 of its 42 references that are in the map. First posted on arXiv 28 Jul 2025.

Builds on

31 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization30 Mar
  2. 2025SkyReels-A1: Expressive Portrait Animation in Video Diffusion Transformers15 Feb
  3. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer1 Dec
  4. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation10 Oct
  5. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency4 Sep

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  5. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  6. 2022Video Diffusion Modelscited by 2,305
  7. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  8. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  9. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  10. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  11. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,081
  12. 2025Qwen2.5-Omni Technical Reportcited by 736
14 more
  1. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  2. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  3. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 350
  4. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  5. 2023AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolationcited by 180
  6. 2025Open-Sora 2.0: Training a Commercial-Level Video Generation Model in 200kcited by 163
  7. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  8. 2024StableAnimator: High-Quality Identity-Preserving Human Image Animationcited by 88
  9. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generationcited by 74
  10. 2024EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animationcited by 72
  11. 2024Sonic: Shifting Focus to Global Audio Perception in Portrait Animationcited by 72
  12. 2025Step-Video-TI2V Technical Report: A State-of-the-Art Text-Driven Image-to-Video Generation Modelcited by 28
  13. 2025EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judgecited by 24
  14. 2022StyleTalker: One-shot Style-based Audio-driven Talking Head Video Generationcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024One Shot, One Talk: Whole-body Talking Avatar from a Single Image
  2. 2024CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attention
  3. 2024DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation
  4. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis
  5. 2025Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generation
  6. 2025LLIA -- Enabling Low-Latency Interactive Avatars: Real-Time Audio-Driven Portrait Video Generation with Diffusion Models
  7. 2022Generating Holistic 3D Human Motion from Speech
  8. 2025A Unit Enhancement and Guidance Framework for Audio-Driven Avatar Video Generation