# High-Performance Reinforcement Learning on Spot: Optimizing Simulation Parameters with Distributional Measures

arXiv: https://arxiv.org/abs/2504.17857 · HTML: https://mlsanity.com/paper/2504.17857/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 23 of its 49 references that are in the map.
- First posted on arXiv 24 Apr 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](https://mlsanity.com/paper/2301.04195/) (2023)
- [Pros and Cons of GAN Evaluation Measures](https://mlsanity.com/paper/1802.03446/) (2018)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [Rapid Locomotion via Reinforcement Learning](https://mlsanity.com/paper/2205.02824/) (2022)
- [ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robots](https://mlsanity.com/paper/2306.14874/) (2023)
- [Training generative neural networks via Maximum Mean Discrepancy optimization](https://mlsanity.com/paper/1505.03906/) (2015)
- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion](https://mlsanity.com/paper/2202.05481/) (2022)
- [Robot Learning from Randomized Simulations: A Review](https://mlsanity.com/paper/2111.00956/) (2021)
- [Sim2Real Transfer for Reinforcement Learning without Dynamics Randomization](https://mlsanity.com/paper/2002.11635/) (2020)
- [Feedback Control For Cassie With Deep Reinforcement Learning](https://mlsanity.com/paper/1803.05580/) (2018)
- [Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations](https://mlsanity.com/paper/2206.11693/) (2022)
- [Hardware as Policy: Mechanical and Computational Co-Optimization using Deep Reinforcement Learning](https://mlsanity.com/paper/2008.04460/) (2020)
- [Learning Inertial Odometry for Dynamic Legged Robot State Estimation](https://mlsanity.com/paper/2111.00789/) (2021)
- [Policy Transfer via Kinematic Domain Randomization and Adaptation](https://mlsanity.com/paper/2011.01891/) (2020)
- [Real-Time Motion Planning of Legged Robots: A Model Predictive Control Approach](https://mlsanity.com/paper/1710.04029/) (2017)
- [Actuator-Constrained Reinforcement Learning for High-Speed Quadrupedal Locomotion](https://mlsanity.com/paper/2312.17507/) (2023)
- [Knowledge Transfer across Imaging Modalities Via Simultaneous Learning of Adaptive Autoencoders for High-Fidelity Mobile Robot Vision](https://mlsanity.com/paper/2105.05010/) (2021)

## Built on by (most cited first, 10 of 10)

- [Learning Massively Multitask World Models for Continuous Control](https://mlsanity.com/paper/2511.19584/) (2025)
- [ZEST: Zero-shot Embodied Skill Transfer for Athletic Robot Control](https://mlsanity.com/paper/2602.00401/) (2026)
- [Versatile Loco-Manipulation through Flexible Interlimb Coordination](https://mlsanity.com/paper/2506.07876/) (2025)
- [Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penalty](https://mlsanity.com/paper/2602.18312/) (2026)
- [Flip Stunts on Bicycle Robots using Iterative Motion Imitation](https://mlsanity.com/paper/2603.27944/) (2026)
- [Differentiable Weightless Controllers: Learning Logic Circuits for Continuous Control](https://mlsanity.com/paper/2512.01467/) (2025)
- [LineRides: Line-Guided Reinforcement Learning for Bicycle Robot Stunts](https://mlsanity.com/paper/2605.05110/) (2026)
- [Too Much of a Good Thing: When sim2real Efforts Impede Policy Learning (And What to Do About It)](https://mlsanity.com/paper/2606.02636/) (2026)
- [Mind the Phase: Effective Rank and Representation Health in Legged Locomotion](https://mlsanity.com/paper/2609.06958/) (2026)
- [Actuator Dynamics Curricula for Narrow-Viability Tasks in Legged Robot Learning](https://mlsanity.com/paper/2609.09492/) (2026)

## Abstract (arXiv)

This work presents an overview of the technical details behind a high performance reinforcement learning policy deployment with the Spot RL Researcher Development Kit for low level motor access on Boston Dynamics Spot. This represents the first public demonstration of an end to end end reinforcement learning policy deployed on Spot hardware with training code publicly available through Nvidia IsaacLab and deployment code available through Boston Dynamics. We utilize Wasserstein Distance and Maximum Mean Discrepancy to quantify the distributional dissimilarity of data collected on hardware and in simulation to measure our sim2real gap. We use these measures as a scoring function for the Covariance Matrix Adaptation Evolution Strategy to optimize simulated parameters that are unknown or difficult to measure from Spot. Our procedure for modeling and training produces high quality reinforcement learning policies capable of multiple gaits, including a flight phase. We deploy policies capable of over 5.2ms locomotion, more than triple Spots default controller maximum speed, robustness to slippery surfaces, disturbance rejection, and overall agility previously unseen on Spot. We detail our method and release our code to support future work on Spot with the low level API.