# Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learning

arXiv: https://arxiv.org/abs/2309.14246 · HTML: https://mlsanity.com/paper/2309.14246/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 55 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 30 of 40)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018)
- [Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics](https://mlsanity.com/paper/2005.04269/) (2020)
- [Legged Locomotion in Challenging Terrains using Egocentric Vision](https://mlsanity.com/paper/2211.07638/) (2022)
- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)
- [The Cramer Distance as a Solution to Biased Wasserstein Gradients](https://mlsanity.com/paper/1705.10743/) (2017)
- [Fully Parameterized Quantile Function for Distributional Reinforcement Learning](https://mlsanity.com/paper/1911.02140/) (2019)
- [Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real World](https://mlsanity.com/paper/2110.05457/) (2021)
- [Risk-Averse Offline Reinforcement Learning](https://mlsanity.com/paper/2102.05371/) (2021)
- [Statistics and Samples in Distributional Reinforcement Learning](https://mlsanity.com/paper/1902.08102/) (2019)
- [Distributional Reinforcement Learning for Efficient Exploration](https://mlsanity.com/paper/1905.06125/) (2019)
- [How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Robotics](https://mlsanity.com/paper/1710.11040/) (2017)
- [Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotion](https://mlsanity.com/paper/2011.02404/) (2020)
- [Worst Cases Policy Gradients](https://mlsanity.com/paper/1911.03618/) (2019)
- [Fast Traversability Estimation for Wild Visual Navigation](https://mlsanity.com/paper/2305.08510/) (2023)
- [Elevation Mapping for Locomotion and Navigation using GPU](https://mlsanity.com/paper/2204.12876/) (2022)

## Built on by (most cited first, 14 of 14)

- [Rethinking Robustness Assessment: Adversarial Attacks on Learning-based Quadrupedal Locomotion Controllers](https://mlsanity.com/paper/2405.12424/) (2024)
- [One Filter to Deploy Them All: Robust Safety for Quadrupedal Navigation in Unknown Environments](https://mlsanity.com/paper/2412.09989/) (2024)
- [Preference Aligned Diffusion Planner for Quadrupedal Locomotion Control](https://mlsanity.com/paper/2410.13586/) (2024)
- [Return Capping: Sample-Efficient CVaR Policy Gradient Optimisation](https://mlsanity.com/paper/2504.20887/) (2025)
- [Bipedalism for Quadrupedal Robots: Versatile Loco-Manipulation through Risk-Adaptive Reinforcement Learning](https://mlsanity.com/paper/2507.20382/) (2025)
- [Myopically Verifiable Probabilistic Certificates for Safe Control and Learning](https://mlsanity.com/paper/2404.16883/) (2024)
- [Scoop-and-Toss: Dynamic Object Collection for Quadrupedal Systems](https://mlsanity.com/paper/2506.09406/) (2025)
- [Distributional Inverse Reinforcement Learning](https://mlsanity.com/paper/2510.03013/) (2025)
- [Risk-Aware Reinforcement Learning with Bandit-Based Adaptation for Quadrupedal Locomotion](https://mlsanity.com/paper/2510.14338/) (2025)
- [Moments Matter:Stabilizing Policy Optimization using Return Distributions](https://mlsanity.com/paper/2601.01803/) (2026)
- [Distributional Active Inference](https://mlsanity.com/paper/2601.20985/) (2026)
- [Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity](https://mlsanity.com/paper/2602.03778/) (2026)
- [Risk-Aware Reinforcement Learning for Mobile Manipulation](https://mlsanity.com/paper/2603.04579/) (2026)
- [Predictive Safety Curricula for Robust Legged Locomotion](https://mlsanity.com/paper/2609.37070/) (2026)

## Abstract (arXiv)

Deployment in hazardous environments requires robots to understand the risks associated with their actions and movements to prevent accidents. Despite its importance, these risks are not explicitly modeled by currently deployed locomotion controllers for legged robots. In this work, we propose a risk sensitive locomotion training method employing distributional reinforcement learning to consider safety explicitly. Instead of relying on a value expectation, we estimate the complete value distribution to account for uncertainty in the robot's interaction with the environment. The value distribution is consumed by a risk metric to extract risk sensitive value estimates. These are integrated into Proximal Policy Optimization (PPO) to derive our method, Distributional Proximal Policy Optimization (DPPO). The risk preference, ranging from risk-averse to risk-seeking, can be controlled by a single parameter, which enables to adjust the robot's behavior dynamically. Importantly, our approach removes the need for additional reward function tuning to achieve risk sensitivity. We show emergent risk sensitive locomotion behavior in simulation and on the quadrupedal robot ANYmal. Videos of the experiments and code are available at https://sites.google.com/leggedrobotics.com/risk-aware-locomotion.