# A study of first-passage time minimization via Q-learning in heated gridworlds

arXiv: https://arxiv.org/abs/2110.02129 · HTML: https://mlsanity.com/paper/2110.02129/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 61 references that are in the map.
- First posted on arXiv 5 Oct 2021.

## Builds on (most cited first, 20 of 20)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon](https://mlsanity.com/paper/1811.06128/) (2018)
- [Machine Learning for Fluid Mechanics](https://mlsanity.com/paper/1905.11075/) (2019)
- [Adversarial Attacks on Neural Network Policies](https://mlsanity.com/paper/1702.02284/) (2017)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [Adversarial Policies: Attacking Deep Reinforcement Learning](https://mlsanity.com/paper/1905.10615/) (2019)
- [Taming the Noise in Reinforcement Learning via Soft Updates](https://mlsanity.com/paper/1512.08562/) (2015)
- [Reinforcement Learning with Perturbed Rewards](https://mlsanity.com/paper/1810.01032/) (2018)
- [Efficient collective swimming by harnessing vortices through deep reinforcement learning](https://mlsanity.com/paper/1802.02674/) (2018)
- [Reinforcement Learning with a Corrupted Reward Channel](https://mlsanity.com/paper/1705.08417/) (2017)
- [Estimating Risk and Uncertainty in Deep Reinforcement Learning](https://mlsanity.com/paper/1905.09638/) (2019)
- [Online Reinforcement Learning in Stochastic Games](https://mlsanity.com/paper/1712.00579/) (2017)
- [Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards](https://mlsanity.com/paper/1906.05841/) (2019)
- [Reinforcement Learning with Combinatorial Actions: An Application to Vehicle Routing](https://mlsanity.com/paper/2010.12001/) (2020)
- [Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signals](https://mlsanity.com/paper/1906.10571/) (2019)
- [Flow Navigation by Smart Microswimmers via Reinforcement Learning](https://mlsanity.com/paper/1701.08848/) (2017)
- [Zermelo's problem: Optimal point-to-point navigation in 2D turbulent flows using Reinforcement Learning](https://mlsanity.com/paper/1907.08591/) (2019)
- [Efficient and Robust Reinforcement Learning with Uncertainty-based Value Expansion](https://mlsanity.com/paper/1912.05328/) (2019)
- [Automatic Risk Adaptation in Distributional Reinforcement Learning](https://mlsanity.com/paper/2106.06317/) (2021)
- [Solving Markov Decision Processes with Reachability Characterization from Mean First Passage Times](https://mlsanity.com/paper/1901.01229/) (2019)

## Abstract (arXiv)

Optimization of first-passage times is required in applications ranging from nanobots navigation to market trading. In such settings, one often encounters unevenly distributed noise levels across the environment. We extensively study how a learning agent fares in 1- and 2- dimensional heated gridworlds with an uneven temperature distribution. The results show certain bias effects in agents trained via simple tabular Q-learning, SARSA, Expected SARSA and Double Q-learning. While high learning rate prevents exploration of regions with higher temperature, low enough rate increases the presence of agents in such regions. The discovered peculiarities and biases of temporal-difference-based reinforcement learning methods should be taken into account in real-world physical applications and agent design.