# Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models

arXiv: https://arxiv.org/abs/2308.15143 · HTML: https://mlsanity.com/paper/2308.15143/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 54 references that are in the map.
- First posted on arXiv 29 Aug 2023.

## Builds on (most cited first, 30 of 30)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [Character Controllers Using Motion VAEs](https://mlsanity.com/paper/2103.14274/) (2021)
- [ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robots](https://mlsanity.com/paper/2306.14874/) (2023)
- [Legged Locomotion in Challenging Terrains using Egocentric Vision](https://mlsanity.com/paper/2211.07638/) (2022)
- [Neural probabilistic motor primitives for humanoid control](https://mlsanity.com/paper/1811.11711/) (2018)
- [From Motor Control to Team Play in Simulated Humanoid Football](https://mlsanity.com/paper/2105.12196/) (2021)
- [Latent Space Policies for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1804.02808/) (2018)
- [Multi-expert learning of adaptive legged locomotion](https://mlsanity.com/paper/2012.05810/) (2020)
- [Learning and Transfer of Modulated Locomotor Controllers](https://mlsanity.com/paper/1610.05182/) (2016)
- [Theory and Experiments on Vector Quantized Autoencoders](https://mlsanity.com/paper/1805.11063/) (2018)
- [Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?](https://mlsanity.com/paper/1611.01055/) (2016)
- [Behavior Priors for Efficient Reinforcement Learning](https://mlsanity.com/paper/2010.14274/) (2020)
- [Barkour: Benchmarking Animal-level Agility with Quadruped Robots](https://mlsanity.com/paper/2305.14654/) (2023)
- [Hierarchical Reinforcement Learning for Precise Soccer Shooting Skills using a Quadrupedal Robot](https://mlsanity.com/paper/2208.01160/) (2022)
- [Neural Categorical Priors for Physics-Based Character Control](https://mlsanity.com/paper/2308.07200/) (2023)
- [Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors](https://mlsanity.com/paper/2203.17138/) (2022)
- [TStarBot-X: An Open-Sourced and Comprehensive Study for Efficient League Training in StarCraft II Full Game](https://mlsanity.com/paper/2011.13729/) (2020)
- [CAJun: Continuous Adaptive Jumping using a Learned Centroidal Controller](https://mlsanity.com/paper/2306.09557/) (2023)
- [TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2011.12895/) (2020)
- [Learning Terrain-Adaptive Locomotion with Agile Behaviors by Imitating Animals](https://mlsanity.com/paper/2308.03273/) (2023)

## Built on by (most cited first, 30 of 37)

- [Whole-body Humanoid Robot Locomotion with Human Reference](https://mlsanity.com/paper/2402.18294/) (2024)
- [Generalized Animal Imitator: Agile Locomotion with Versatile Motion Prior](https://mlsanity.com/paper/2310.01408/) (2023)
- [RobotKeyframing: Learning Locomotion with High-Level Objectives via Mixture of Dense and Sparse Rewards](https://mlsanity.com/paper/2407.11562/) (2024)
- [Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models](https://mlsanity.com/paper/2407.02666/) (2024)
- [Humanoid Whole-Body Locomotion on Narrow Terrain via Dynamic Balance and Reinforcement Learning](https://mlsanity.com/paper/2502.17219/) (2025)
- [SPECI: Skill Prompts based Hierarchical Continual Imitation Learning for Robot Manipulation](https://mlsanity.com/paper/2504.15561/) (2025)
- [Adapt On-the-Go: Behavior Modulation for Single-Life Robot Deployment](https://mlsanity.com/paper/2311.01059/) (2023)
- [Preference Aligned Diffusion Planner for Quadrupedal Locomotion Control](https://mlsanity.com/paper/2410.13586/) (2024)
- [Gait-Conditioned Reinforcement Learning with Multi-Phase Curriculum for Humanoid Locomotion](https://mlsanity.com/paper/2505.20619/) (2025)
- [Motion Priors Reimagined: Adapting Flat-Terrain Skills for Complex Quadruped Mobility](https://mlsanity.com/paper/2505.16084/) (2025)
- [Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systems](https://mlsanity.com/paper/2512.24385/) (2025)
- [AcL: Action Learner for Fault-Tolerant Quadruped Locomotion Control](https://mlsanity.com/paper/2503.21401/) (2025)
- [An Efficient Model-Based Approach on Learning Agile Motor Skills without Reinforcement](https://mlsanity.com/paper/2403.01962/) (2024)
- [Reinforcement Learning from Wild Animal Videos](https://mlsanity.com/paper/2412.04273/) (2024)
- [AINav: Large Language Model-Based Adaptive Interactive Navigation](https://mlsanity.com/paper/2503.22942/) (2025)
- [Feature-Based vs. GAN-Based Learning from Demonstrations: When and Why](https://mlsanity.com/paper/2507.05906/) (2025)
- [End-to-End Crop Row Navigation via LiDAR-Based Deep Reinforcement Learning](https://mlsanity.com/paper/2509.18608/) (2025)
- [Learning Terrain-Specialized Policies for Adaptive Locomotion in Challenging Environments](https://mlsanity.com/paper/2509.20635/) (2025)
- [Agile perceptive multi-skill locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2607.13579/) (2026)
- [HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object-Interaction](https://mlsanity.com/paper/2608.16222/) (2026)
- [Unlock Reliable Skill Inference for Quadruped Adaptive Behavior by Skill Graph](https://mlsanity.com/paper/2311.06015/) (2023)
- [Integrating Controllable Motion Skills from Demonstrations](https://mlsanity.com/paper/2408.03018/) (2024)
- [Coordinating Spinal and Limb Dynamics for Enhanced Sprawling Robot Mobility](https://mlsanity.com/paper/2504.14103/) (2025)
- [Topology-Aware and Highly Generalizable Deep Reinforcement Learning for Efficient Retrieval in Multi-Deep Storage Systems](https://mlsanity.com/paper/2506.14787/) (2025)
- [INGRID: Intelligent Generative Robotic Design Using Large Language Models](https://mlsanity.com/paper/2509.03842/) (2025)
- [Dynamic Adaptive Legged Locomotion Policy via Decoupling Reaction Force Control and Gait Control](https://mlsanity.com/paper/2509.13737/) (2025)
- [Unlocking the Potential of Soft Actor-Critic for Imitation Learning](https://mlsanity.com/paper/2509.24539/) (2025)
- [Plasma Shape Control via Zero-shot Generative Reinforcement Learning](https://mlsanity.com/paper/2510.17531/) (2025)
- [Co-jump: Cooperative Jumping with Quadrupedal Robots via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2602.10514/) (2026)
- [Cooperative-Competitive Team Play of Real-World Craft Robots](https://mlsanity.com/paper/2602.21119/) (2026)

## Abstract (arXiv)

Knowledge from animals and humans inspires robotic innovations. Numerous efforts have been made to achieve agile locomotion in quadrupedal robots through classical controllers or reinforcement learning approaches. These methods usually rely on physical models or handcrafted rewards to accurately describe the specific system, rather than on a generalized understanding like animals do. Here we propose a hierarchical framework to construct primitive-, environmental- and strategic-level knowledge that are all pre-trainable, reusable and enrichable for legged robots. The primitive module summarizes knowledge from animal motion data, where, inspired by large pre-trained models in language and image understanding, we introduce deep generative models to produce motor control signals stimulating legged robots to act like real animals. Then, we shape various traversing capabilities at a higher level to align with the environment by reusing the primitive module. Finally, a strategic module is trained focusing on complex downstream tasks by reusing the knowledge from previous levels. We apply the trained hierarchical controllers to the MAX robot, a quadrupedal robot developed in-house, to mimic animals, traverse complex obstacles and play in a designed challenging multi-agent chase tag game, where lifelike agility and strategy emerge in the robots.