# The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settings

arXiv: https://arxiv.org/abs/2308.12646 · HTML: https://mlsanity.com/paper/2308.12646/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 59 references that are in the map.
- First posted on arXiv 24 Aug 2023.

## Builds on (most cited first, 20 of 20)

- [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020)
- [Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models](https://mlsanity.com/paper/2211.09707/) (2022)
- [BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis](https://mlsanity.com/paper/2203.05297/) (2022)
- [Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots](https://mlsanity.com/paper/1810.12541/) (2018)
- [DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models](https://mlsanity.com/paper/2305.04919/) (2023)
- [Gesticulator: A framework for semantically-aware speech-driven gesture generation](https://mlsanity.com/paper/2001.09326/) (2020)
- [A Comprehensive Review of Data-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2301.05339/) (2023)
- [Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning](https://mlsanity.com/paper/2108.00262/) (2021)
- [The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generation](https://mlsanity.com/paper/2208.10441/) (2022)
- [QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation](https://mlsanity.com/paper/2305.11094/) (2023)
- [A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020](https://mlsanity.com/paper/2102.11617/) (2021)
- [Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings](https://mlsanity.com/paper/2006.09888/) (2020)
- [Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022](https://mlsanity.com/paper/2303.08737/) (2023)
- [Integrated Speech and Gesture Synthesis](https://mlsanity.com/paper/2108.11436/) (2021)
- [HEMVIP: Human Evaluation of Multiple Videos in Parallel](https://mlsanity.com/paper/2101.11898/) (2021)
- [Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis](https://mlsanity.com/paper/2306.09417/) (2023)
- [To Rate or Not To Rate: Investigating Evaluation Methods for Generated Co-Speech Gestures](https://mlsanity.com/paper/2108.05709/) (2021)
- [Passing a Non-verbal Turing Test: Evaluating Gesture Animations Generated from Speech](https://mlsanity.com/paper/2107.00712/) (2021)
- [SGToolkit: An Interactive Gesture Authoring Toolkit for Embodied Conversational Agents](https://mlsanity.com/paper/2108.04636/) (2021)
- [MPE4G: Multimodal Pretrained Encoder for Co-Speech Gesture Generation](https://mlsanity.com/paper/2305.15740/) (2023)

## Built on by (most cited first, 28 of 28)

- [Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model](https://mlsanity.com/paper/2404.01862/) (2024)
- [Dyadic Interaction Modeling for Social Behavior Generation](https://mlsanity.com/paper/2403.09069/) (2024)
- [Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representation](https://mlsanity.com/paper/2309.05455/) (2023)
- [MotionScript: Natural Language Descriptions for Expressive 3D Human Motions](https://mlsanity.com/paper/2312.12634/) (2023)
- [Freetalker: Controllable Speech and Text-Driven Gesture Generation Based on Diffusion Models for Enhanced Speaker Naturalness](https://mlsanity.com/paper/2401.03476/) (2024)
- [UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons](https://mlsanity.com/paper/2309.07051/) (2023)
- [The DiffuseStyleGesture+ entry to the GENEA Challenge 2023](https://mlsanity.com/paper/2308.13879/) (2023)
- [Beyond Talking -- Generating Holistic 3D Human Dyadic Motion for Communication](https://mlsanity.com/paper/2403.19467/) (2024)
- [Co3Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion](https://mlsanity.com/paper/2505.01746/) (2025)
- [Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agents](https://mlsanity.com/paper/2510.04637/) (2025)
- [Unified speech and gesture synthesis using flow matching](https://mlsanity.com/paper/2310.05181/) (2023)
- [EMOTION: Expressive Motion Sequence Generation for Humanoid Robots with In-Context Learning](https://mlsanity.com/paper/2410.23234/) (2024)
- [HOP: Heterogeneous Topology-based Multimodal Entanglement for Co-Speech Gesture Generation](https://mlsanity.com/paper/2503.01175/) (2025)
- [Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis](https://mlsanity.com/paper/2404.19622/) (2024)
- [Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesis](https://mlsanity.com/paper/2410.06327/) (2024)
- [Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents](https://mlsanity.com/paper/2408.04127/) (2024)
- [InterAct: A Large-Scale Dataset of Dynamic, Expressive and Interactive Activities between Two People in Daily Scenarios](https://mlsanity.com/paper/2509.05747/) (2025)
- [DiM-Gestor: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2](https://mlsanity.com/paper/2411.16729/) (2024)
- [Gesture Evaluation in Virtual Reality](https://mlsanity.com/paper/2509.12816/) (2025)
- [Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmark](https://mlsanity.com/paper/2511.01233/) (2025)
- [LLAniMAtion: LLAMA Driven Gesture Animation](https://mlsanity.com/paper/2405.08042/) (2024)
- [Integrating Representational Gestures into Automatically Generated Embodied Explanations and its Effects on Understanding and Interaction Quality](https://mlsanity.com/paper/2406.12544/) (2024)
- [2D or not 2D: How Does the Dimensionality of Gesture Representation Affect 3D Co-Speech Gesture Generation?](https://mlsanity.com/paper/2409.10357/) (2024)
- [A Survey of Body and Face Motion: Datasets, Performance Evaluation Metrics and Generative Techniques](https://mlsanity.com/paper/2512.09005/) (2025)
- [Evaluation of Generative Models for Emotional 3D Animation Generation in VR](https://mlsanity.com/paper/2512.16081/) (2025)
- [Prompt-to-Gesture: Measuring the Capabilities of Image-to-Video Deictic Gesture Generation](https://mlsanity.com/paper/2604.14953/) (2026)
- [The GENEA Challenge 2026: A Large-Scale Disentangled Evaluation of Speech-Driven Gesture Generation on the Seamless Interaction Dataset](https://mlsanity.com/paper/2608.10839/) (2026)
- [A Benchmark for Spatially Grounded Gesture Generation](https://mlsanity.com/paper/2610.03105/) (2026)

## Abstract (arXiv)

This paper reports on the GENEA Challenge 2023, in which participating teams built speech-driven gesture-generation systems using the same speech and motion dataset, followed by a joint evaluation. This year's challenge provided data on both sides of a dyadic interaction, allowing teams to generate full-body motion for an agent given its speech (text and audio) and the speech and motion of the interlocutor. We evaluated 12 submissions and 2 baselines together with held-out motion-capture data in several large-scale user studies. The studies focused on three aspects: 1) the human-likeness of the motion, 2) the appropriateness of the motion for the agent's own speech whilst controlling for the human-likeness of the motion, and 3) the appropriateness of the motion for the behaviour of the interlocutor in the interaction, using a setup that controls for both the human-likeness of the motion and the agent's own speech. We found a large span in human-likeness between challenge submissions, with a few systems rated close to human mocap. Appropriateness seems far from being solved, with most submissions performing in a narrow range slightly above chance, far behind natural motion. The effect of the interlocutor is even more subtle, with submitted systems at best performing barely above chance. Interestingly, a dyadic system being highly appropriate for agent speech does not necessarily imply high appropriateness for the interlocutor. Additional material is available via the project website at https://svito-zar.github.io/GENEAchallenge2023/ .