# Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions

arXiv: https://arxiv.org/abs/2203.15103 · HTML: https://mlsanity.com/paper/2203.15103/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 57 references that are in the map.
- First posted on arXiv 28 Mar 2022.

## Builds on (most cited first, 30 of 31)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [Imitation Learning as f-Divergence Minimization](https://mlsanity.com/paper/1905.12888/) (2019)
- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Composition](https://mlsanity.com/paper/2011.01387/) (2020)
- [GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning](https://mlsanity.com/paper/1810.05762/) (2018)
- [Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion](https://mlsanity.com/paper/2202.05481/) (2022)
- [Robust Imitation of Diverse Behaviors](https://mlsanity.com/paper/1707.02747/) (2017)
- [Learning human behaviors from motion capture by adversarial imitation](https://mlsanity.com/paper/1707.02201/) (2017)
- [Learning Symmetric and Low-energy Locomotion](https://mlsanity.com/paper/1801.08093/) (2018)
- [Fast and Efficient Locomotion via Learned Gait Transitions](https://mlsanity.com/paper/2104.04644/) (2021)
- [Coupling Vision and Proprioception for Navigation of Legged Robots](https://mlsanity.com/paper/2112.02094/) (2021)
- [Policies Modulating Trajectory Generators](https://mlsanity.com/paper/1910.02812/) (2019)
- [Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learning](https://mlsanity.com/paper/1901.07517/) (2019)
- [3D Simulation for Robot Arm Control with Deep Q-Learning](https://mlsanity.com/paper/1609.03759/) (2016)
- [Learning Agile Locomotion Skills with a Mentor](https://mlsanity.com/paper/2011.05541/) (2020)

## Built on by (most cited first, 30 of 89)

- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Open-TeleVision: Teleoperation with Immersive Active Visual Feedback](https://mlsanity.com/paper/2407.01512/) (2024)
- [Expressive Whole-Body Control for Humanoid Robots](https://mlsanity.com/paper/2402.16796/) (2024)
- [Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control](https://mlsanity.com/paper/2401.16889/) (2024)
- [ExBody2: Advanced Expressive Humanoid Whole-Body Control](https://mlsanity.com/paper/2412.13196/) (2024)
- [PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interaction](https://mlsanity.com/paper/2312.04393/) (2023)
- [Visual Whole-Body Control for Legged Loco-Manipulation](https://mlsanity.com/paper/2403.16967/) (2024)
- [Mobile-TeleVision: Predictive Motion Priors for Humanoid Whole-Body Control](https://mlsanity.com/paper/2412.07773/) (2024)
- [AMO: Adaptive Motion Optimization for Hyper-Dexterous Humanoid Whole-Body Control](https://mlsanity.com/paper/2505.03738/) (2025)
- [DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasets](https://mlsanity.com/paper/2404.19264/) (2024)
- [Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotion](https://mlsanity.com/paper/2308.12517/) (2023)
- [Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations](https://mlsanity.com/paper/2206.11693/) (2022)
- [Language Reward Modulation for Pretraining Reinforcement Learning](https://mlsanity.com/paper/2308.12270/) (2023)
- [Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Response](https://mlsanity.com/paper/2312.11460/) (2023)
- [Robust and Versatile Bipedal Jumping Control through Reinforcement Learning](https://mlsanity.com/paper/2302.09450/) (2023)
- [Learning and Adapting Agile Locomotion Skills by Transferring Experience](https://mlsanity.com/paper/2304.09834/) (2023)
- [Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors](https://mlsanity.com/paper/2203.17138/) (2022)
- [Fleet-DAgger: Interactive Robot Fleet Learning with Scalable Human Supervision](https://mlsanity.com/paper/2206.14349/) (2022)
- [RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.04150/) (2023)
- [Whole-body Humanoid Robot Locomotion with Human Reference](https://mlsanity.com/paper/2402.18294/) (2024)
- [RL + Model-based Control: Using On-demand Optimal Control to Learn Versatile Legged Locomotion](https://mlsanity.com/paper/2305.17842/) (2023)
- [Creating a Dynamic Quadrupedal Robotic Goalkeeper with Reinforcement Learning](https://mlsanity.com/paper/2210.04435/) (2022)
- [CaT: Constraints as Terminations for Legged Locomotion Reinforcement Learning](https://mlsanity.com/paper/2403.18765/) (2024)
- [Curriculum-Based Reinforcement Learning for Quadrupedal Jumping: A Reference-free Design](https://mlsanity.com/paper/2401.16337/) (2024)
- [RobotKeyframing: Learning Locomotion with High-Level Objectives via Mixture of Dense and Sparse Rewards](https://mlsanity.com/paper/2407.11562/) (2024)
- [World Model-based Perception for Visual Legged Locomotion](https://mlsanity.com/paper/2409.16784/) (2024)
- [Design and Control of a Bipedal Robotic Character](https://mlsanity.com/paper/2501.05204/) (2025)
- [PhysHSI: Towards a Real-World Generalizable and Natural Humanoid-Scene Interaction System](https://mlsanity.com/paper/2510.11072/) (2025)
- [A Learning Framework for Diverse Legged Robot Locomotion Using Barrier-Based Style Rewards](https://mlsanity.com/paper/2409.15780/) (2024)

## Abstract (arXiv)

Training a high-dimensional simulated agent with an under-specified reward function often leads the agent to learn physically infeasible strategies that are ineffective when deployed in the real world. To mitigate these unnatural behaviors, reinforcement learning practitioners often utilize complex reward functions that encourage physically plausible behaviors. However, a tedious labor-intensive tuning process is often required to create hand-designed rewards which might not easily generalize across platforms and tasks. We propose substituting complex reward functions with "style rewards" learned from a dataset of motion capture demonstrations. A learned style reward can be combined with an arbitrary task reward to train policies that perform tasks using naturalistic strategies. These natural strategies can also facilitate transfer to the real world. We build upon Adversarial Motion Priors -- an approach from the computer graphics domain that encodes a style reward from a dataset of reference motions -- to demonstrate that an adversarial approach to training policies can produce behaviors that transfer to a real quadrupedal robot without requiring complex reward functions. We also demonstrate that an effective style reward can be learned from a few seconds of motion capture data gathered from a German Shepherd and leads to energy-efficient locomotion strategies with natural gait transitions.