# Dealing with uncertainty: balancing exploration and exploitation in deep recurrent reinforcement learning

arXiv: https://arxiv.org/abs/2310.08331 · HTML: https://mlsanity.com/paper/2310.08331/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 39 references that are in the map.
- First posted on arXiv 12 Oct 2023.

## Builds on (most cited first, 12 of 12)

- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Multimodal End-to-End Autonomous Driving](https://mlsanity.com/paper/1906.03199/) (2019)
- [Playing FPS Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05521/) (2016)
- [Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control](https://mlsanity.com/paper/1909.09884/) (2019)
- [MADRaS : Multi Agent Driving Simulator](https://mlsanity.com/paper/2010.00993/) (2020)
- [Deep Q-Learning based Reinforcement Learning Approach for Network Intrusion Detection](https://mlsanity.com/paper/2111.13978/) (2021)
- [Decision-making Strategy on Highway for Autonomous Vehicles using Deep Reinforcement Learning](https://mlsanity.com/paper/2007.08691/) (2020)
- [Behavioral decision-making for urban autonomous driving in the presence of pedestrians using Deep Recurrent Q-Network](https://mlsanity.com/paper/2010.13407/) (2020)
- [Autonomous quadrotor obstacle avoidance based on dueling double deep recurrent Q-learning with monocular vision](https://mlsanity.com/paper/2002.03510/) (2020)
- [ε-BMC: A Bayesian Ensemble Approach to Epsilon-Greedy Exploration in Model-Free Reinforcement Learning](https://mlsanity.com/paper/2007.00869/) (2020)
- [Deep Recurrent Q-Learning vs Deep Q-Learning on a simple Partially Observable Markov Decision Process with Minecraft](https://mlsanity.com/paper/1903.04311/) (2019)

## Built on by (most cited first, 1 of 1)

- [Neural Nonmyopic Bayesian Optimization in Dynamic Cost Settings](https://mlsanity.com/paper/2601.06505/) (2026)

## Abstract (arXiv)

Incomplete knowledge of the environment leads an agent to make decisions under uncertainty. One of the major dilemmas in Reinforcement Learning (RL) where an autonomous agent has to balance two contrasting needs in making its decisions is: exploiting the current knowledge of the environment to maximize the cumulative reward as well as exploring actions that allow improving the knowledge of the environment, hopefully leading to higher reward values (exploration-exploitation trade-off). Concurrently, another relevant issue regards the full observability of the states, which may not be assumed in all applications. For instance, when 2D images are considered as input in an RL approach used for finding the best actions within a 3D simulation environment. In this work, we address these issues by deploying and testing several techniques to balance exploration and exploitation trade-off on partially observable systems for predicting steering wheels in autonomous driving scenarios. More precisely, the final aim is to investigate the effects of using both adaptive and deterministic exploration strategies coupled with a Deep Recurrent Q-Network. Additionally, we adapted and evaluated the impact of a modified quadratic loss function to improve the learning phase of the underlying Convolutional Recurrent Neural Network. We show that adaptive methods better approximate the trade-off between exploration and exploitation and, in general, Softmax and Max-Boltzmann strategies outperform epsilon-greedy techniques.