# Imitation and Adaptation Based on Consistency: A Quadruped Robot Imitates Animals from Videos Using Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2203.05973 · HTML: https://mlsanity.com/paper/2203.05973/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 2 Mar 2022.

## Builds on (most cited first, 18 of 18)

- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping](https://mlsanity.com/paper/1709.07857/) (2017)
- [Multi-expert learning of adaptive legged locomotion](https://mlsanity.com/paper/2012.05810/) (2020)
- [RGBD-Dog: Predicting Canine Pose from RGBD Sensors](https://mlsanity.com/paper/2004.07788/) (2020)
- [Robust Quadruped Jumping via Deep Reinforcement Learning](https://mlsanity.com/paper/2011.07089/) (2020)
- [Multi-Layered Safety for Legged Robots via Control Barrier Functions and Model Predictive Control](https://mlsanity.com/paper/2011.00032/) (2020)
- [SFV: Reinforcement Learning of Physical Skills from Videos](https://mlsanity.com/paper/1810.03599/) (2018)
- [Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning](https://mlsanity.com/paper/2104.05043/) (2021)
- [Iterative Reinforcement Learning Based Design of Dynamic Locomotion Skills for Cassie](https://mlsanity.com/paper/1903.09537/) (2019)
- [Zero-Shot Terrain Generalization for Visual Locomotion Policies](https://mlsanity.com/paper/2011.05513/) (2020)
- [MPC-based Controller with Terrain Insight for Dynamic Legged Locomotion](https://mlsanity.com/paper/1909.13842/) (2019)
- [Fast Trajectory Optimization for Legged Robots using Vertex-based ZMP Constraints](https://mlsanity.com/paper/1705.10313/) (2017)

## Built on by (most cited first, 4 of 4)

- [Learning Adaptive Hydrodynamic Models Using Neural ODEs in Complex Conditions](https://mlsanity.com/paper/2410.00490/) (2024)
- [Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos](https://mlsanity.com/paper/2410.09286/) (2024)
- [Reinforcement Learning from Wild Animal Videos](https://mlsanity.com/paper/2412.04273/) (2024)
- [CopyQuad: Quadrupedal Skill Transfer from a Single Side-View Video Without a Depth Model](https://mlsanity.com/paper/2610.16737/) (2026)

## Abstract (arXiv)

The essence of quadrupeds' movements is the movement of the center of gravity, which has a pattern in the action of quadrupeds. However, the gait motion planning of the quadruped robot is time-consuming. Animals in nature can provide a large amount of gait information for robots to learn and imitate. Common methods learn animal posture with a motion capture system or numerous motion data points. In this paper, we propose a video imitation adaptation network (VIAN) that can imitate the action of animals and adapt it to the robot from a few seconds of video. The deep learning model extracts key points during animal motion from videos. The VIAN eliminates noise and extracts key information of motion with a motion adaptor, and then applies the extracted movements function as the motion pattern into deep reinforcement learning (DRL). To ensure similarity between the learning result and the animal motion in the video, we introduce rewards that are based on the consistency of the motion. DRL explores and learns to maintain balance from movement patterns from videos, imitates the action of animals, and eventually, allows the model to learn the gait or skills from short motion videos of different animals and to transfer the motion pattern to the real robot.