# Physics-informed Imitative Reinforcement Learning for Real-world Driving

arXiv: https://arxiv.org/abs/2407.02508 · HTML: https://mlsanity.com/paper/2407.02508/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving.
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Placed in

- End-to-end autonomous driving / [Imitation-learning end-to-end planners](https://mlsanity.com/driving/imitation-learning-end-to-end-planners/)
- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 30 of 30)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [Parting with Misconceptions about Learning-based Vehicle Motion Planning](https://mlsanity.com/paper/2306.07962/) (2023)
- [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://mlsanity.com/paper/2006.14480/) (2020)
- [Multi-Game Decision Transformers](https://mlsanity.com/paper/2205.15241/) (2022)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research](https://mlsanity.com/paper/2310.08710/) (2023)
- [Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios](https://mlsanity.com/paper/2212.11419/) (2022)
- [A Survey of Deep Learning Applications to Autonomous Vehicle Control](https://mlsanity.com/paper/1912.10773/) (2019)
- [Port-Hamiltonian Neural Networks for Learning Explicit Time-Dependent Dynamical Systems](https://mlsanity.com/paper/2107.08024/) (2021)
- [Elastic Decision Transformer](https://mlsanity.com/paper/2307.02484/) (2023)
- [GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Driving](https://mlsanity.com/paper/2111.08575/) (2021)
- [Dissipative Hamiltonian Neural Networks: Learning Dissipative and Conservative Dynamics Separately](https://mlsanity.com/paper/2201.10085/) (2022)
- [Hyper-Decision Transformer for Efficient Online Policy Adaptation](https://mlsanity.com/paper/2304.08487/) (2023)
- [Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targets](https://mlsanity.com/paper/2306.14069/) (2023)
- [Monocular Depth Estimation using Multi-Scale Continuous CRFs as Sequential Deep Networks](https://mlsanity.com/paper/1803.00891/) (2018)
- [Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration](https://mlsanity.com/paper/2309.09408/) (2023)
- [Moving SLAM: Fully Unsupervised Deep Learning in Non-Rigid Scenes](https://mlsanity.com/paper/2105.02195/) (2021)
- [Bridging Imitation and Online Reinforcement Learning: An Optimistic Tale](https://mlsanity.com/paper/2303.11369/) (2023)
- [Combining RL and IL using a dynamic, performance-based modulation over learning signals and its application to local planning](https://mlsanity.com/paper/2405.09760/) (2024)

## Built on by (most cited first, 2 of 2)

- [Autonomous Vehicle Path Planning by Searching With Differentiable Simulation](https://mlsanity.com/paper/2511.11043/) (2025)
- [Physics-informed Diffusion Mamba Transformer for Real-world Driving](https://mlsanity.com/paper/2602.00808/) (2026)

## Abstract (arXiv)

Recent advances in imitative reinforcement learning (IRL) have considerably enhanced the ability of autonomous agents to assimilate expert demonstrations, leading to rapid skill acquisition in a range of demanding tasks. However, such learning-based agents face significant challenges when transferring knowledge to highly dynamic closed-loop environments. Their performance is significantly impacted by the conflicting optimization objectives of imitation learning (IL) and reinforcement learning (RL), sample inefficiency, and the complexity of uncovering the hidden world model and physics. To address this challenge, we propose a physics-informed IRL that is entirely data-driven. It leverages both expert demonstration data and exploratory data with a joint optimization objective, allowing the underlying physical principles of vehicle dynamics to emerge naturally from the training process. The performance is evaluated through empirical experiments and results exceed popular IL, RL and IRL algorithms in closed-loop settings on Waymax benchmark. Our approach exhibits 37.8% reduction in collision rate and 22.2% reduction in off-road rate compared to the baseline method.