# GAN-based Reactive Motion Synthesis with Class-aware Discriminators for Human-human Interaction

arXiv: https://arxiv.org/abs/2110.00380 · HTML: https://mlsanity.com/paper/2110.00380/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 53 references that are in the map.
- First posted on arXiv 1 Oct 2021.

## Builds on (most cited first, 22 of 22)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [Coupled Generative Adversarial Networks](https://mlsanity.com/paper/1606.07536/) (2016)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [HP-GAN: Probabilistic 3D human motion prediction via GAN](https://mlsanity.com/paper/1711.09561/) (2017)
- [Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition](https://mlsanity.com/paper/1607.07043/) (2016)
- [QuaterNet: A Quaternion-based Recurrent Model for Human Motion](https://mlsanity.com/paper/1805.06485/) (2018)
- [Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation](https://mlsanity.com/paper/1804.06055/) (2018)
- [Semi-Supervised Learning with Generative Adversarial Networks](https://mlsanity.com/paper/1606.01583/) (2016)
- [Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks](https://mlsanity.com/paper/1707.05740/) (2017)
- [Action Recognition using Visual Attention](https://mlsanity.com/paper/1511.04119/) (2015)
- [Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning](https://mlsanity.com/paper/1805.02335/) (2018)
- [Modeling Human Motion with Quaternion-based Neural Networks](https://mlsanity.com/paper/1901.07677/) (2019)
- [Spatio-temporal Manifold Learning for Human Motions via Long-horizon Modeling](https://mlsanity.com/paper/1908.07214/) (2019)
- [Structure-Aware Human-Action Generation](https://mlsanity.com/paper/2007.01971/) (2020)
- [Learning Social Affordance for Human-Robot Interaction](https://mlsanity.com/paper/1604.03692/) (2016)
- [Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model](https://mlsanity.com/paper/2010.09950/) (2020)
- [Skeleton-aided Articulated Motion Generation](https://mlsanity.com/paper/1707.01058/) (2017)
- [GlocalNet: Class-aware Long-term Human Motion Synthesis](https://mlsanity.com/paper/2012.10744/) (2020)

## Built on by (most cited first, 18 of 18)

- [ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactions](https://mlsanity.com/paper/2311.17057/) (2023)
- [Interactive Character Control with Auto-Regressive Motion Diffusion Models](https://mlsanity.com/paper/2306.00416/) (2023)
- [Music-Driven Group Choreography](https://mlsanity.com/paper/2303.12337/) (2023)
- [Ready-to-React: Online Reaction Policy for Two-Character Interaction Generation](https://mlsanity.com/paper/2502.20370/) (2025)
- [A Survey on Human Interaction Motion Generation](https://mlsanity.com/paper/2503.12763/) (2025)
- [DROP: Dynamics Responses from Human Motion Prior and Projective Dynamics](https://mlsanity.com/paper/2309.13742/) (2023)
- [3D Human Interaction Generation: A Survey](https://mlsanity.com/paper/2503.13120/) (2025)
- [Think-Then-React: Towards Unconstrained Human Action-to-Reaction Generation](https://mlsanity.com/paper/2503.16451/) (2025)
- [Interaction Mix and Match: Synthesizing Close Interaction using Conditional Hierarchical GAN with Multi-Hot Class Embedding](https://mlsanity.com/paper/2208.00774/) (2022)
- [Interaction-aware Joint Attention Estimation Using People Attributes](https://mlsanity.com/paper/2308.05382/) (2023)
- [Unified Multi-Modal Interactive & Reactive 3D Motion Generation via Rectified Flow](https://mlsanity.com/paper/2509.24099/) (2025)
- [Two-Person Interaction Augmentation with Skeleton Priors](https://mlsanity.com/paper/2404.05490/) (2024)
- [MDD: A Dataset for Text-and-Music Conditioned Duet Dance Generation](https://mlsanity.com/paper/2508.16911/) (2025)
- [ExGes: Expressive Human Motion Retrieval and Modulation for Audio-Driven Gesture Synthesis](https://mlsanity.com/paper/2503.06499/) (2025)
- [PlanMoGPT: Flow-Enhanced Progressive Planning for Text to Motion Synthesis](https://mlsanity.com/paper/2506.17912/) (2025)
- [Adversarial Attention for Human Motion Synthesis](https://mlsanity.com/paper/2204.11751/) (2022)
- [Fatigue-PINN: Physics-Informed Fatigue-Driven Motion Modulation and Synthesis](https://mlsanity.com/paper/2502.19056/) (2025)
- [JAX-IK: Real-Time Inverse Kinematics for Generating Multi-Constrained Movements of Virtual Human Characters](https://mlsanity.com/paper/2507.00792/) (2025)

## Abstract (arXiv)

Creating realistic characters that can react to the users' or another character's movement can benefit computer graphics, games and virtual reality hugely. However, synthesizing such reactive motions in human-human interactions is a challenging task due to the many different ways two humans can interact. While there are a number of successful researches in adapting the generative adversarial network (GAN) in synthesizing single human actions, there are very few on modelling human-human interactions. In this paper, we propose a semi-supervised GAN system that synthesizes the reactive motion of a character given the active motion from another character. Our key insights are two-fold. First, to effectively encode the complicated spatial-temporal information of a human motion, we empower the generator with a part-based long short-term memory (LSTM) module, such that the temporal movement of different limbs can be effectively modelled. We further include an attention module such that the temporal significance of the interaction can be learned, which enhances the temporal alignment of the active-reactive motion pair. Second, as the reactive motion of different types of interactions can be significantly different, we introduce a discriminator that not only tells if the generated movement is realistic or not, but also tells the class label of the interaction. This allows the use of such labels in supervising the training of the generator. We experiment with the SBU and the HHOI datasets. The high quality of the synthetic motion demonstrates the effective design of our generator, and the discriminability of the synthesis also demonstrates the strength of our discriminator.