# Reward Training Wheels: Adaptive Auxiliary Rewards for Robotics Reinforcement Learning

arXiv: https://arxiv.org/abs/2503.15724 · HTML: https://mlsanity.com/paper/2503.15724/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 19 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 16 of 16)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [Learning by Playing - Solving Sparse Reward Tasks from Scratch](https://mlsanity.com/paper/1802.10567/) (2018)
- [Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexity](https://mlsanity.com/paper/2210.09579/) (2022)
- [BaRC: Backward Reachability Curriculum for Robotic Reinforcement Learning](https://mlsanity.com/paper/1806.06161/) (2018)
- [Benchmarking Metric Ground Navigation](https://mlsanity.com/paper/2008.13315/) (2020)
- [Benchmarking Reinforcement Learning Techniques for Autonomous Navigation](https://mlsanity.com/paper/2210.04839/) (2022)
- [Toward Wheeled Mobility on Vertically Challenging Terrain: Platforms, Datasets, and Algorithms](https://mlsanity.com/paper/2303.00998/) (2023)
- [Reinforcement Learning for Wheeled Mobility on Vertically Challenging Terrain](https://mlsanity.com/paper/2409.02383/) (2024)
- [Grounded Curriculum Learning](https://mlsanity.com/paper/2409.19816/) (2024)
- [Curriculum Reinforcement Learning for Complex Reward Functions](https://mlsanity.com/paper/2410.16790/) (2024)
- [Informativeness of Reward Functions in Reinforcement Learning](https://mlsanity.com/paper/2402.07019/) (2024)
- [VertiSelector: Automatic Curriculum Learning for Wheeled Mobility on Vertically Challenging Terrain](https://mlsanity.com/paper/2409.17469/) (2024)

## Built on by (most cited first, 4 of 4)

- [ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problems](https://mlsanity.com/paper/2510.07151/) (2025)
- [DREAM: Domain-aware Reasoning for Efficient Autonomous Underwater Monitoring](https://mlsanity.com/paper/2509.13666/) (2025)
- [APPLV: Adaptive Planner Parameter Learning from Vision-Language-Action Model](https://mlsanity.com/paper/2603.08862/) (2026)
- [A Unified Framework for Dynamic Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/2608.08158/) (2026)

## Abstract (arXiv)

Robotics Reinforcement Learning (RL) often relies on carefully engineered auxiliary rewards to supplement sparse primary learning objectives to compensate for the lack of large-scale, real-world, trial-and-error data. While these auxiliary rewards accelerate learning, they require significant engineering effort, may introduce human biases, and cannot adapt to the robot's evolving capabilities during training. In this paper, we introduce Reward Training Wheels (RTW), a teacher-student framework that automates auxiliary reward adaptation for robotics RL. To be specific, the RTW teacher dynamically adjusts auxiliary reward weights based on the student's evolving capabilities to determine which auxiliary reward aspects require more or less emphasis to improve the primary objective. We demonstrate RTW on two challenging robot tasks: navigation in highly constrained spaces and off-road vehicle mobility on vertically challenging terrain. In simulation, RTW outperforms expert-designed rewards by 2.35% in navigation success rate and improves off-road mobility performance by 122.62%, while achieving 35% and 3X faster training efficiency, respectively. Physical robot experiments further validate RTW's effectiveness, achieving a perfect success rate (5/5 trials vs. 2/5 for expert-designed rewards) and improving vehicle stability with up to 47.4% reduction in orientation angles.