# Reactive Reinforcement Learning in Asynchronous Environments

arXiv: https://arxiv.org/abs/1802.06139 · HTML: https://mlsanity.com/paper/1802.06139/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Feb 2018.

## Built on by (most cited first, 12 of 12)

- [Asynchronous Reinforcement Learning for Real-Time Control of Physical Robots](https://mlsanity.com/paper/2203.12759/) (2022)
- [Asynchronous Multi-Agent Reinforcement Learning for Efficient Real-Time Multi-Robot Cooperative Exploration](https://mlsanity.com/paper/2301.03398/) (2023)
- [Real-Time Reasoning Agents in Evolving Environments](https://mlsanity.com/paper/2511.04898/) (2025)
- [Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference](https://mlsanity.com/paper/2412.14355/) (2024)
- [Handling Delay in Real-Time Reinforcement Learning](https://mlsanity.com/paper/2503.23478/) (2025)
- [Chanakya: Learning Runtime Decisions for Adaptive Real-Time Perception](https://mlsanity.com/paper/2106.05665/) (2021)
- [Reducing the Cost of Cycle-Time Tuning for Real-World Policy Optimization](https://mlsanity.com/paper/2305.05760/) (2023)
- [Finding the Time to Think: Learning Planning Budgets in Real-Time RL](https://mlsanity.com/paper/2606.26463/) (2026)
- [Temporally Layered Architecture for Adaptive, Distributed and Continuous Control](https://mlsanity.com/paper/2301.00723/) (2022)
- [Physical Atari: A Robust and Accessible Platform for Real-time Reinforcement Learning on Robots](https://mlsanity.com/paper/2606.19357/) (2026)
- [A comparison of RL-based and PID controllers for 6-DOF swimming robots: hybrid underwater object tracking](https://mlsanity.com/paper/2401.16618/) (2024)
- [Synchronous vs Asynchronous Reinforcement Learning in a Real World Robot](https://mlsanity.com/paper/2503.14554/) (2025)

## Abstract (arXiv)

The relationship between a reinforcement learning (RL) agent and an asynchronous environment is often ignored. Frequently used models of the interaction between an agent and its environment, such as Markov Decision Processes (MDP) or Semi-Markov Decision Processes (SMDP), do not capture the fact that, in an asynchronous environment, the state of the environment may change during computation performed by the agent. In an asynchronous environment, minimizing reaction time---the time it takes for an agent to react to an observation---also minimizes the time in which the state of the environment may change following observation. In many environments, the reaction time of an agent directly impacts task performance by permitting the environment to transition into either an undesirable terminal state or a state where performing the chosen action is inappropriate. We propose a class of reactive reinforcement learning algorithms that address this problem of asynchronous environments by immediately acting after observing new state information. We compare a reactive SARSA learning algorithm with the conventional SARSA learning algorithm on two asynchronous robotic tasks (emergency stopping and impact prevention), and show that the reactive RL algorithm reduces the reaction time of the agent by approximately the duration of the algorithm's learning update. This new class of reactive algorithms may facilitate safer control and faster decision making without any change to standard learning guarantees.