# The ReprGesture entry to the GENEA Challenge 2022

arXiv: https://arxiv.org/abs/2208.12133 · HTML: https://mlsanity.com/paper/2208.12133/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 21 references that are in the map.
- First posted on arXiv 25 Aug 2022.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Domain Separation Networks](https://mlsanity.com/paper/1608.06019/) (2016)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020)
- [BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis](https://mlsanity.com/paper/2203.05297/) (2022)
- [Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots](https://mlsanity.com/paper/1810.12541/) (2018)
- [Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders](https://mlsanity.com/paper/2108.06720/) (2021)
- [Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation](https://mlsanity.com/paper/2203.13161/) (2022)
- [Gesticulator: A framework for semantically-aware speech-driven gesture generation](https://mlsanity.com/paper/2001.09326/) (2020)
- [Analyzing Input and Output Representations for Speech-Driven Gesture Generation](https://mlsanity.com/paper/1903.03369/) (2019)
- [The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generation](https://mlsanity.com/paper/2208.10441/) (2022)
- [A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020](https://mlsanity.com/paper/2102.11617/) (2021)
- [Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation](https://mlsanity.com/paper/2007.09170/) (2020)
- [HEMVIP: Human Evaluation of Multiple Videos in Parallel](https://mlsanity.com/paper/2101.11898/) (2021)
- [Freeform Body Motion Generation from Speech](https://mlsanity.com/paper/2203.02291/) (2022)

## Built on by (most cited first, 9 of 9)

- [DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models](https://mlsanity.com/paper/2305.04919/) (2023)
- [MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models](https://mlsanity.com/paper/2403.09471/) (2024)
- [QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation](https://mlsanity.com/paper/2305.11094/) (2023)
- [Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022](https://mlsanity.com/paper/2303.08737/) (2023)
- [Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Control](https://mlsanity.com/paper/2312.15900/) (2023)
- [UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons](https://mlsanity.com/paper/2309.07051/) (2023)
- [The DiffuseStyleGesture+ entry to the GENEA Challenge 2023](https://mlsanity.com/paper/2308.13879/) (2023)
- [Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference](https://mlsanity.com/paper/2403.10805/) (2024)
- [A conversational gesture synthesis system based on emotions and semantics](https://mlsanity.com/paper/2507.03147/) (2025)

## Abstract (arXiv)

This paper describes the ReprGesture entry to the Generation and Evaluation of Non-verbal Behaviour for Embodied Agents (GENEA) challenge 2022. The GENEA challenge provides the processed datasets and performs crowdsourced evaluations to compare the performance of different gesture generation systems. In this paper, we explore an automatic gesture generation system based on multimodal representation learning. We use WavLM features for audio, FastText features for text and position and rotation matrix features for gesture. Each modality is projected to two distinct subspaces: modality-invariant and modality-specific. To learn inter-modality-invariant commonalities and capture the characters of modality-specific representations, gradient reversal layer based adversarial classifier and modality reconstruction decoders are used during training. The gesture decoder generates proper gestures using all representations and features related to the rhythm in the audio. Our code, pre-trained models and demo are available at https://github.com/YoungSeng/ReprGesture.