# Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022

arXiv: https://arxiv.org/abs/2303.08737 · HTML: https://mlsanity.com/paper/2303.08737/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 152 references that are in the map.
- First posted on arXiv 15 Mar 2023.

## Builds on (most cited first, 30 of 43)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020)
- [Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models](https://mlsanity.com/paper/2211.09707/) (2022)
- [BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis](https://mlsanity.com/paper/2203.05297/) (2022)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots](https://mlsanity.com/paper/1810.12541/) (2018)
- [Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation](https://mlsanity.com/paper/2203.13161/) (2022)
- [Gesticulator: A framework for semantically-aware speech-driven gesture generation](https://mlsanity.com/paper/2001.09326/) (2020)
- [A Comprehensive Review of Data-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2301.05339/) (2023)
- [Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning](https://mlsanity.com/paper/2108.00262/) (2021)
- [Analyzing Input and Output Representations for Speech-Driven Gesture Generation](https://mlsanity.com/paper/1903.03369/) (2019)
- [The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generation](https://mlsanity.com/paper/2208.10441/) (2022)
- [A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020](https://mlsanity.com/paper/2102.11617/) (2021)
- [Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings](https://mlsanity.com/paper/2210.01448/) (2022)
- [Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings](https://mlsanity.com/paper/2006.09888/) (2020)
- [Deep Learning Based Assessment of Synthetic Speech Naturalness](https://mlsanity.com/paper/2104.11673/) (2021)
- [A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents](https://mlsanity.com/paper/2101.03769/) (2021)
- [DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model](https://mlsanity.com/paper/2301.10047/) (2023)
- [Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation](https://mlsanity.com/paper/2007.09170/) (2020)

## Built on by (most cited first, 29 of 29)

- [The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settings](https://mlsanity.com/paper/2308.12646/) (2023)
- [UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons](https://mlsanity.com/paper/2309.07051/) (2023)
- [The DiffuseStyleGesture+ entry to the GENEA Challenge 2023](https://mlsanity.com/paper/2308.13879/) (2023)
- [Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis](https://mlsanity.com/paper/2306.09417/) (2023)
- [Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Models](https://mlsanity.com/paper/2507.20220/) (2025)
- [Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agents](https://mlsanity.com/paper/2510.04637/) (2025)
- [Unified speech and gesture synthesis using flow matching](https://mlsanity.com/paper/2310.05181/) (2023)
- [Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference](https://mlsanity.com/paper/2403.10805/) (2024)
- [Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony](https://mlsanity.com/paper/2408.09397/) (2024)
- [Towards the generation of synchronized and believable non-verbal facial behaviors of a talking virtual agent](https://mlsanity.com/paper/2311.12804/) (2023)
- [Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis](https://mlsanity.com/paper/2404.19622/) (2024)
- [LLM Gesticulator: Leveraging Large Language Models for Scalable and Controllable Co-Speech Gesture Synthesis](https://mlsanity.com/paper/2410.10851/) (2024)
- [Establishing a Unified Evaluation Framework for Human Motion Generation: A Comparative Analysis of Metrics](https://mlsanity.com/paper/2405.07680/) (2024)
- [Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesis](https://mlsanity.com/paper/2410.06327/) (2024)
- [DiM-Gesture: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2 framework](https://mlsanity.com/paper/2408.00370/) (2024)
- [DiM-Gestor: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2](https://mlsanity.com/paper/2411.16729/) (2024)
- [Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmark](https://mlsanity.com/paper/2511.01233/) (2025)
- [DuoGesture: Motion-Grounded Semantic Conditioning and Biomechanical Beat Priors for Co-Speech Gesture Generation](https://mlsanity.com/paper/2605.26236/) (2026)
- [Audio is all in one: speech-driven gesture synthetics using WavLM pre-trained model](https://mlsanity.com/paper/2308.05995/) (2023)
- [TranSTYLer: Multimodal Behavioral Style Transfer for Facial and Body Gestures Generation](https://mlsanity.com/paper/2308.10843/) (2023)
- [Integrating Representational Gestures into Automatically Generated Embodied Explanations and its Effects on Understanding and Interaction Quality](https://mlsanity.com/paper/2406.12544/) (2024)
- [Mitigation of gender bias in automatic facial non-verbal behaviors generation](https://mlsanity.com/paper/2410.07274/) (2024)
- [ReCoM: Realistic Co-Speech Motion Generation with Recurrent Embedded Transformer](https://mlsanity.com/paper/2503.21847/) (2025)
- [ImaGGen: Zero-Shot Generation of Co-Speech Semantic Gestures Grounded in Language and Image Input](https://mlsanity.com/paper/2510.17617/) (2025)
- [Prompt-to-Gesture: Measuring the Capabilities of Image-to-Video Deictic Gesture Generation](https://mlsanity.com/paper/2604.14953/) (2026)
- [SiGnature: Explicit Motion Diffusion for Stylized Semantic Gesture](https://mlsanity.com/paper/2606.15889/) (2026)
- [The GENEA Challenge 2026: A Large-Scale Disentangled Evaluation of Speech-Driven Gesture Generation on the Seamless Interaction Dataset](https://mlsanity.com/paper/2608.10839/) (2026)
- [Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue](https://mlsanity.com/paper/2609.04250/) (2026)
- [GestAdapt: Workspace-Conditioned Co-Speech Gesture Generation for Humanoid Robots](https://mlsanity.com/paper/2609.38400/) (2026)

## Abstract (arXiv)

This paper reports on the second GENEA Challenge to benchmark data-driven automatic co-speech gesture generation. Participating teams used the same speech and motion dataset to build gesture-generation systems. Motion generated by all these systems was rendered to video using a standardised visualisation pipeline and evaluated in several large, crowdsourced user studies. Unlike when comparing different research papers, differences in results are here only due to differences between methods, enabling direct comparison between systems. The dataset was based on 18 hours of full-body motion capture, including fingers, of different persons engaging in a dyadic conversation. Ten teams participated in the challenge across two tiers: full-body and upper-body gesticulation. For each tier, we evaluated both the human-likeness of the gesture motion and its appropriateness for the specific speech signal. Our evaluations decouple human-likeness from gesture appropriateness, which has been a difficult problem in the field. The evaluation results show some synthetic gesture conditions being rated as significantly more human-like than 3D human motion capture. To the best of our knowledge, this has not been demonstrated before. On the other hand, all synthetic motion is found to be vastly less appropriate for the speech than the original motion-capture recordings. We also find that conventional objective metrics do not correlate well with subjective human-likeness ratings in this large evaluation. The one exception is the Fr'echet gesture distance (FGD), which achieves a Kendall's tau rank correlation of around -0.5. Based on the challenge results we formulate numerous recommendations for system building and evaluation.