# Safe Model-based Off-policy Reinforcement Learning for Eco-Driving in Connected and Automated Hybrid Electric Vehicles

arXiv: https://arxiv.org/abs/2105.11640 · HTML: https://mlsanity.com/paper/2105.11640/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 19 of its 52 references that are in the map.
- First posted on arXiv 25 May 2021.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)

## Builds on (most cited first, 19 of 19)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [Benchmarking Model-Based Reinforcement Learning](https://mlsanity.com/paper/1907.02057/) (2019)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [Practical Reinforcement Learning For MPC: Learning from sparse objectives in under an hour on a real robot](https://mlsanity.com/paper/2003.03200/) (2020)
- [ABC-LMPC: Safe Sample-Based Learning MPC for Stochastic Nonlinear Dynamical Systems with Adjustable Boundary Conditions](https://mlsanity.com/paper/2003.01410/) (2020)
- [Sample-Based Learning Model Predictive Control for Linear Uncertain Systems](https://mlsanity.com/paper/1904.06432/) (2019)
- [A Deep Reinforcement Learning Framework for Eco-driving in Connected and Automated Hybrid Electric Vehicles](https://mlsanity.com/paper/2101.05372/) (2021)
- [A GPU Implementation of a Look-Ahead Optimal Controller for Eco-Driving Based on Dynamic Programming](https://mlsanity.com/paper/2104.01284/) (2021)

## Built on by (most cited first, 7 of 7)

- [Mitigating Metropolitan Carbon Emissions with Dynamic Eco-driving at Scale](https://mlsanity.com/paper/2408.05609/) (2024)
- [Generative Models and Connected and Automated Vehicles: A Survey in Exploring the Intersection of Transportation and AI](https://mlsanity.com/paper/2403.10559/) (2024)
- [Progression Cognition Reinforcement Learning with Prioritized Experience for Multi-Vehicle Pursuit](https://mlsanity.com/paper/2306.05016/) (2023)
- [Eco-Driving Control of Connected and Automated Vehicles using Neural Network based Rollout](https://mlsanity.com/paper/2310.10878/) (2023)
- [S4TP: Social-Suitable and Safety-Sensitive Trajectory Planning for Autonomous Vehicles](https://mlsanity.com/paper/2404.11946/) (2024)
- [Platoon-Centric Green Light Optimal Speed Advisory Using Safe Reinforcement Learning](https://mlsanity.com/paper/2509.12378/) (2025)
- [Safe and Sustainable Electric Bus Charging Scheduling with Constrained Hierarchical DRL](https://mlsanity.com/paper/2512.03059/) (2025)

## Abstract (arXiv)

Connected and Automated Hybrid Electric Vehicles have the potential to reduce fuel consumption and travel time in real-world driving conditions. The eco-driving problem seeks to design optimal speed and power usage profiles based upon look-ahead information from connectivity and advanced mapping features. Recently, Deep Reinforcement Learning (DRL) has been applied to the eco-driving problem. While the previous studies synthesize simulators and model-free DRL to reduce online computation, this work proposes a Safe Off-policy Model-Based Reinforcement Learning algorithm for the eco-driving problem. The advantages over the existing literature are three-fold. First, the combination of off-policy learning and the use of a physics-based model improves the sample efficiency. Second, the training does not require any extrinsic rewarding mechanism for constraint satisfaction. Third, the feasibility of trajectory is guaranteed by using a safe set approximated by deep generative models. The performance of the proposed method is benchmarked against a baseline controller representing human drivers, a previously designed model-free DRL strategy, and the wait-and-see optimal solution. In simulation, the proposed algorithm leads to a policy with a higher average speed and a better fuel economy compared to the model-free agent. Compared to the baseline controller, the learned strategy reduces the fuel consumption by more than 21% while keeping the average speed comparable.