mlsanityData to 14 Oct 2026

Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesis

arXiv 2410.06327 ↗First posted 8 Oct 2024cs.HC cs.CV cs.GRcited by 3

Not on the map. Why?

Abstract (arXiv)

Current evaluation practices in speech-driven gesture generation lack standardisation and focus on aspects that are easy to measure over aspects that actually matter. This leads to a situation where it is impossible to know what is the state of the art, or to know which method works better for which purpose when comparing two publications. In this position paper, we review and give details on issues with existing gesture-generation evaluation, and present a novel proposal for remedying them. Specifically, we announce an upcoming living leaderboard to benchmark progress in conversational motion synthesis. Unlike earlier gesture-generation challenges, the leaderboard will be updated with large-scale user studies of new gesture-generation systems multiple times per year, and systems on the leaderboard can be submitted to any publication venue that their authors prefer. By evolving the leaderboard evaluation data and tasks over time, the effort can keep driving progress towards the most important end goals identified by the community. We actively seek community involvement across the entire evaluation pipeline: from data and tasks for the evaluation, via tooling, to the systems evaluated. In other words, our proposal will not only make it easier for researchers to perform good evaluations, but their collective input and contributions will also help drive the future of gesture-generation research.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 46 of its 83 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

46 of 83 references.

References · 46

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  3. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  4. 2022EDGE: Editable Dance Generation From Musiccited by 287
  5. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  6. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 202
  7. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 190
  8. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  9. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 178
  10. 2019Effectively Unbiased FID and Inception Score and where to find themcited by 165
  11. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 158
  12. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 129
34 more
  1. 2021DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformercited by 111
  2. 2018Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robotscited by 110
  3. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  4. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 107
  5. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 105
  6. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
  7. 2020Gesticulator: A framework for semantically-aware speech-driven gesture generationcited by 84
  8. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 84
  9. 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82
  10. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  11. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 62
  12. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  13. 2022The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generationcited by 40
  14. 2023QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generationcited by 40
  15. 2023LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generationcited by 40
  16. 2024Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesiscited by 40
  17. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39
  18. 2021A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020cited by 38
  19. 2020Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settingscited by 36
  20. 2023Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022cited by 29
  21. 2023The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settingscited by 28
  22. 2023Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representationcited by 28
  23. 2021A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agentscited by 24
  24. 2023DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Modelcited by 22
  25. 2023BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformercited by 21
  26. 2023Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generationcited by 17
  27. 2021Integrated Speech and Gesture Synthesiscited by 16
  28. 2021HEMVIP: Human Evaluation of Multiple Videos in Parallelcited by 13
  29. 2023Investigating Range-Equalizing Bias in Mean Opinion Score Ratings of Synthesized Speechcited by 13
  30. 2021To Rate or Not To Rate: Investigating Evaluation Methods for Generated Co-Speech Gesturescited by 12
  31. 2021Passing a Non-verbal Turing Test: Evaluating Gesture Animations Generated from Speechcited by 11
  32. 2021SGToolkit: An Interactive Gesture Authoring Toolkit for Embodied Conversational Agentscited by 11
  33. 2023Why We Should Report the Details in Subjective Evaluation of TTS More Rigorouslycited by 10
  34. 2022Deep Gesture Generation for Social Robots Using Type-Specific Librariescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2024Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesiscited by 26
  2. 2025A conversational gesture synthesis system based on emotions and semanticscited by 0
  3. 2026Perceptually Grounded and Semantics-Aware Evaluation for Holistic Co-Speech Gesture Generationcited by 0