# TADPO: Reinforcement Learning Goes Off-road

arXiv: https://arxiv.org/abs/2603.05995 · HTML: https://mlsanity.com/paper/2603.05995/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RL for autonomous driving and traffic, one of 15 directions in Deep reinforcement learning methods.
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 6 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 30 of 31)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018)
- [Planning with Goal-Conditioned Policies](https://mlsanity.com/paper/1911.08453/) (2019)
- [Goal-Conditioned Reinforcement Learning with Imagined Subgoals](https://mlsanity.com/paper/2107.00541/) (2021)
- [Model Predictive Path Integral Control using Covariance Variable Importance Sampling](https://mlsanity.com/paper/1509.01149/) (2015)
- [V-STRONG: Visual Self-Supervised Traversability Learning for Off-road Navigation](https://mlsanity.com/paper/2312.16016/) (2023)
- [Safe Driving via Expert Guided Policy Optimization](https://mlsanity.com/paper/2110.06831/) (2021)
- [How Does It Feel? Self-Supervised Costmap Learning for Off-Road Vehicle Traversability](https://mlsanity.com/paper/2209.10788/) (2022)
- [RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving](https://mlsanity.com/paper/2402.19341/) (2024)
- [CTS: Concurrent Teacher-Student Reinforcement Learning for Legged Locomotion](https://mlsanity.com/paper/2405.10830/) (2024)
- [Guided Exploration with Proximal Policy Optimization using a Single Demonstration](https://mlsanity.com/paper/2007.03328/) (2020)
- [Learning Risk-Aware Costmaps via Inverse Reinforcement Learning for Off-Road Navigation](https://mlsanity.com/paper/2302.00134/) (2023)
- [Off-road Autonomous Vehicles Traversability Analysis and Trajectory Planning Based on Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1909.06953/) (2019)
- [Model Predictive Control for Aggressive Driving Over Uneven Terrain](https://mlsanity.com/paper/2311.12284/) (2023)
- [Proximal Policy Distillation](https://mlsanity.com/paper/2407.15134/) (2024)
- [Learning Terrain-Aware Kinodynamic Model for Autonomous Off-Road Rally Driving With Model Predictive Path Integral Control](https://mlsanity.com/paper/2305.00676/) (2023)
- [Deep Reinforcement Learning with Enhanced PPO for Safe Mobile Robot Navigation](https://mlsanity.com/paper/2405.16266/) (2024)
- [SALON: Self-supervised Adaptive Learning for Off-road Navigation](https://mlsanity.com/paper/2412.07826/) (2024)
- [Long Range Navigator (LRN): Extending robot planning horizons beyond metric maps](https://mlsanity.com/paper/2504.13149/) (2025)
- [Learning from demonstrations with SACR2: Soft Actor-Critic with Reward Relabeling](https://mlsanity.com/paper/2110.14464/) (2021)
- [WROOM: An Autonomous Driving Approach for Off-Road Navigation](https://mlsanity.com/paper/2404.08855/) (2024)
- [Predictive Barrier Lyapunov Function Based Control for Safe Trajectory Tracking of an Aerial Manipulator](https://mlsanity.com/paper/2212.04625/) (2022)

## Built on by (most cited first, 3 of 3)

- [TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations](https://mlsanity.com/paper/2606.17386/) (2026)
- [MILER: Semantic Mid-Level Representation for Sim-to-Real Reinforcement Learning in Unstructured Autonomous Driving](https://mlsanity.com/paper/2609.20747/) (2026)
- [Verti-WM: A Physics-Aided Exteroceptive World Model for Off-Road Reinforcement Learning](https://mlsanity.com/paper/2609.23118/) (2026)

## Abstract (arXiv)

Off-road autonomous driving poses significant challenges such as navigating unmapped, variable terrain with uncertain and diverse dynamics. Addressing these challenges requires effective long-horizon planning and adaptable control. Reinforcement Learning (RL) offers a promising solution by learning control policies directly from interaction. However, because off-road driving is a long-horizon task with low-signal rewards, standard RL methods are challenging to apply in this setting. We introduce TADPO, a novel policy gradient formulation that extends Proximal Policy Optimization (PPO), leveraging off-policy trajectories for teacher guidance and on-policy trajectories for student exploration. Building on this, we develop a vision-based, end-to-end RL system for high-speed off-road driving, capable of navigating extreme slopes and obstacle-rich terrain. We demonstrate our performance in simulation and, importantly, zero-shot sim-to-real transfer on a full-scale off-road vehicle. To our knowledge, this work represents the first deployment of RL-based policies on a full-scale off-road platform.