# Benchmarking Potential Based Rewards for Learning Humanoid Locomotion

arXiv: https://arxiv.org/abs/2307.10142 · HTML: https://mlsanity.com/paper/2307.10142/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 19 Jul 2023.

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study](https://mlsanity.com/paper/2006.05990/) (2020)
- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Composition](https://mlsanity.com/paper/2011.01387/) (2020)
- [Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning](https://mlsanity.com/paper/2105.08328/) (2021)
- [The MIT Humanoid Robot: Design, Motion Planning, and Control For Acrobatic Behaviors](https://mlsanity.com/paper/2104.09025/) (2021)
- [Automatic LQR Tuning Based on Gaussian Process Global Optimization](https://mlsanity.com/paper/1605.01950/) (2016)
- [Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning](https://mlsanity.com/paper/2208.06721/) (2022)
- [Bayesian Optimization Using Domain Knowledge on the ATRIAS Biped](https://mlsanity.com/paper/1709.06047/) (2017)
- [Learning Spring Mass Locomotion: Guiding Policies with a Reduced-Order Model](https://mlsanity.com/paper/2010.11234/) (2020)
- [Learning to Brachiate via Simplified Model Imitation](https://mlsanity.com/paper/2205.03943/) (2022)
- [Learning to Run with Potential-Based Reward Shaping and Demonstrations from Video Data](https://mlsanity.com/paper/2012.08824/) (2020)

## Built on by (most cited first, 15 of 15)

- [Whole-body Humanoid Robot Locomotion with Human Reference](https://mlsanity.com/paper/2402.18294/) (2024)
- [Learning Whole-Body Loco-Manipulation for Omni-Directional Task Space Pose Tracking with a Wheeled-Quadrupedal-Manipulator](https://mlsanity.com/paper/2412.03012/) (2024)
- [Distillation-PPO: A Novel Two-Stage Reinforcement Learning Framework for Humanoid Robot Perceptive Locomotion](https://mlsanity.com/paper/2503.08299/) (2025)
- [Natural Humanoid Robot Locomotion with Generative Motion Prior](https://mlsanity.com/paper/2503.09015/) (2025)
- [Fully Spiking Neural Network for Legged Robots](https://mlsanity.com/paper/2310.05022/) (2023)
- [CLF-RL: Control Lyapunov Function Guided Reinforcement Learning](https://mlsanity.com/paper/2508.09354/) (2025)
- [Structural Optimization of Lightweight Bipedal Robot via SERL](https://mlsanity.com/paper/2408.15632/) (2024)
- [Residual MPC: Blending Reinforcement Learning with GPU-Parallelized Model Predictive Control](https://mlsanity.com/paper/2510.12717/) (2025)
- [Efficient Reinforcement Learning for Jumping Monopods](https://mlsanity.com/paper/2309.07038/) (2023)
- [Boosting Reinforcement Learning Algorithms in Continuous Robotic Reaching Tasks using Adaptive Potential Functions](https://mlsanity.com/paper/2402.04581/) (2024)
- [ECO: Energy-Constrained Optimization with Reinforcement Learning for Humanoid Walking](https://mlsanity.com/paper/2602.06445/) (2026)
- [MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins](https://mlsanity.com/paper/2606.26392/) (2026)
- [PTRL: Prior Transfer Deep Reinforcement Learning for Legged Robots Locomotion](https://mlsanity.com/paper/2504.05629/) (2025)
- [Adaptive Smooth Tchebycheff Attention for Multi-Objective Policy Optimization](https://mlsanity.com/paper/2605.12771/) (2026)
- [Locally-Guided Actor-Critic: Training a Goal-conditioned Actor with a Subgoal-aware Critic](https://mlsanity.com/paper/2608.30406/) (2026)

## Abstract (arXiv)

The main challenge in developing effective reinforcement learning (RL) pipelines is often the design and tuning the reward functions. Well-designed shaping reward can lead to significantly faster learning. Naively formulated rewards, however, can conflict with the desired behavior and result in overfitting or even erratic performance if not properly tuned. In theory, the broad class of potential based reward shaping (PBRS) can help guide the learning process without affecting the optimal policy. Although several studies have explored the use of potential based reward shaping to accelerate learning convergence, most have been limited to grid-worlds and low-dimensional systems, and RL in robotics has predominantly relied on standard forms of reward shaping. In this paper, we benchmark standard forms of shaping with PBRS for a humanoid robot. We find that in this high-dimensional system, PBRS has only marginal benefits in convergence speed. However, the PBRS reward terms are significantly more robust to scaling than typical reward shaping approaches, and thus easier to tune.