# Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control

arXiv: https://arxiv.org/abs/1806.08666 · HTML: https://mlsanity.com/paper/1806.08666/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2018.

## Built on by (most cited first, 30 of 39)

- [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021)
- [Character Controllers Using Motion VAEs](https://mlsanity.com/paper/2103.14274/) (2021)
- [Synthesis of Compositional Animations from Textual Descriptions](https://mlsanity.com/paper/2103.14675/) (2021)
- [AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanism](https://mlsanity.com/paper/2309.00796/) (2023)
- [Synthesizing Diverse Human Motions in 3D Indoor Scenes](https://mlsanity.com/paper/2305.12411/) (2023)
- [Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model](https://mlsanity.com/paper/2304.08577/) (2023)
- [CIRCLE: Capture In Rich Contextual Environments](https://mlsanity.com/paper/2303.17912/) (2023)
- [The Wanderings of Odysseus in 3D Scenes](https://mlsanity.com/paper/2112.09251/) (2021)
- [Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structure](https://mlsanity.com/paper/2111.12159/) (2021)
- [GUESS:GradUally Enriching SyntheSis for Text-Driven Human Motion Generation](https://mlsanity.com/paper/2401.02142/) (2024)
- [Action-conditioned On-demand Motion Generation](https://mlsanity.com/paper/2207.08164/) (2022)
- [TrajeVAE: Controllable Human Motion Generation from Trajectories](https://mlsanity.com/paper/2104.00351/) (2021)
- [Graph-based Normalizing Flow for Human Motion Generation and Reconstruction](https://mlsanity.com/paper/2104.03020/) (2021)
- [DivaTrack: Diverse Bodies and Motions from Acceleration-Enhanced Three-Point Trackers](https://mlsanity.com/paper/2402.09211/) (2024)
- [AAMDM: Accelerated Auto-regressive Motion Diffusion Model](https://mlsanity.com/paper/2401.06146/) (2023)
- [Diffusion-based Human Motion Style Transfer with Semantic Guidance](https://mlsanity.com/paper/2405.06646/) (2024)
- [MOVIN: Real-time Motion Capture using a Single LiDAR](https://mlsanity.com/paper/2309.09314/) (2023)
- [DivDiff: A Conditional Diffusion Model for Diverse Human Motion Prediction](https://mlsanity.com/paper/2409.00014/) (2024)
- [Diffusion Implicit Policy for Unpaired Scene-aware Motion Synthesis](https://mlsanity.com/paper/2412.02261/) (2024)
- [Incorporating Transformer and LSTM to Kalman Filter with EM algorithm for state estimation](https://mlsanity.com/paper/2105.00250/) (2021)
- [Pose Representations for Deep Skeletal Animation](https://mlsanity.com/paper/2111.13907/) (2021)
- [SpeechAct: Towards Generating Whole-body Motion from Speech](https://mlsanity.com/paper/2311.17425/) (2023)
- [HMVLM: Human Motion-Vision-Lanuage Model via MoE LoRA](https://mlsanity.com/paper/2511.01463/) (2025)
- [NEURAL MARIONETTE: A Transformer-based Multi-action Human Motion Synthesis System](https://mlsanity.com/paper/2209.13204/) (2022)
- [Controllable Motion Synthesis and Reconstruction with Autoregressive Diffusion Models](https://mlsanity.com/paper/2304.04681/) (2023)
- [MOST: Motion Diffusion Model for Rare Text via Temporal Clip Banzhaf Interaction](https://mlsanity.com/paper/2507.06590/) (2025)
- [Diverse Motion In-betweening with Dual Posture Stitching](https://mlsanity.com/paper/2303.14457/) (2023)
- [Jointly Understand Your Command and Intention:Reciprocal Co-Evolution between Scene-Aware 3D Human Motion Synthesis and Analysis](https://mlsanity.com/paper/2503.00371/) (2025)
- [Dynamical Deep Generative Latent Modeling of 3D Skeletal Motion](https://mlsanity.com/paper/2106.10393/) (2021)
- [Adversarial Attention for Human Motion Synthesis](https://mlsanity.com/paper/2204.11751/) (2022)

## Abstract (arXiv)

This paper introduces a new generative deep learning network for human motion synthesis and control. Our key idea is to combine recurrent neural networks (RNNs) and adversarial training for human motion modeling. We first describe an efficient method for training a RNNs model from prerecorded motion data. We implement recurrent neural networks with long short-term memory (LSTM) cells because they are capable of handling nonlinear dynamics and long term temporal dependencies present in human motions. Next, we train a refiner network using an adversarial loss, similar to Generative Adversarial Networks (GANs), such that the refined motion sequences are indistinguishable from real motion capture data using a discriminative network. We embed contact information into the generative deep learning model to further improve the performance of our generative model. The resulting model is appealing to motion synthesis and control because it is compact, contact-aware, and can generate an infinite number of naturally looking motions with infinite lengths. Our experiments show that motions generated by our deep learning model are always highly realistic and comparable to high-quality motion capture data. We demonstrate the power and effectiveness of our models by exploring a variety of applications, ranging from random motion synthesis, online/offline motion control, and motion filtering. We show the superiority of our generative model by comparison against baseline models.