mlsanityData to 7 Oct 2026

Real-Time Reinforcement Learning

arXiv 1911.04448 ↗First posted 11 Nov 2019cs.LG stat.MLcited by 38

Not on the map. Why?

Abstract (arXiv)

Markov Decision Processes (MDPs), the mathematical framework underlying most algorithms in Reinforcement Learning (RL), are often used in a way that wrongfully assumes that the state of an agent's environment does not change during action selection. As RL systems based on MDPs begin to find application in real-world safety critical situations, this mismatch between the assumptions underlying classical MDPs and the reality of real-time computation may lead to undesirable outcomes. In this paper, we introduce a new framework, in which states and actions evolve simultaneously and show how it is related to the classical MDP formulation. We analyze existing algorithms under the new real-time formulation and show why they are suboptimal when used in real-time. We then use those insights to create a new algorithm Real-Time Actor-Critic (RTAC) that outperforms the existing state-of-the-art continuous control algorithm Soft Actor-Critic both in real-time and non-real-time settings. Code and videos can be found at https://github.com/rmst/rtrl.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 5; 2023: 4; 2024: 11; 2025: 3; 2026: 10.

38 papers.

Sort:

By area:

  1. 2021Acting in Delayed Environments with Non-Stationary Markov Policiescited by 28
  2. 2021Revisiting State Augmentation methods for Reinforcement Learning with Stochastic Delayscited by 20
  3. 2021Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomizationcited by 18
  4. 2022Real-Time Reinforcement Learning for Vision-Based Robotics Utilizing Local and Remote Computerscited by 13
  5. 2025One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learningcited by 9
  6. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learningcited by 7
  7. 2024Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inferencecited by 7
  8. 2024DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delayscited by 6
  9. 2021Chanakya: Learning Runtime Decisions for Adaptive Real-Time Perceptioncited by 5
  10. 2023Reducing the Cost of Cycle-Time Tuning for Real-World Policy Optimizationcited by 4
  11. 2023DiAReL: Reinforcement Learning with Disturbance Awareness for Robust Sim2Real Policy Transfer in Robot Controlcited by 4
  12. 2023R^3: On-device Real-Time Deep Reinforcement Learning for Autonomous Roboticscited by 4
26 more
  1. 2024Deployable Reinforcement Learning with Variable Control Ratecited by 4
  2. 2021What Robot do I Need? Fast Co-Adaptation of Morphology and Control using Graph Neural Networkscited by 2
  3. 2023Facilitating Sim-to-real by Intrinsic Stochasticity of Real-Time Simulation in Reinforcement Learning for Robot Manipulationcited by 2
  4. 2024Reinforcement Learning with Elastic Time Stepscited by 2
  5. 2024Pausing Policy Learning in Non-stationary Reinforcement Learningcited by 2
  6. 2024Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path Planningcited by 2
  7. 2026Finding the Time to Think: Learning Planning Budgets in Real-Time RLcited by 2
  8. 2022REPTILE: A Proactive Real-Time Deep Reinforcement Learning Self-adaptive Frameworkcited by 1
  9. 2022Off-Beat Multi-Agent Reinforcement Learningcited by 1
  10. 2022Temporally Layered Architecture for Adaptive, Distributed and Continuous Controlcited by 1
  11. 2024Control in Stochastic Environment with Delays: A Model-based Reinforcement Learning Approachcited by 1
  12. 2026Reinforcement Learning for Control Systems with Time Delays: A Comprehensive Surveycited by 1
  13. 2026Physical Atari: A Robust and Accessible Platform for Real-time Reinforcement Learning on Robotscited by 1
  14. 2022Machine Learning based Anomaly Detection for Smart Shirt: A Systematic Reviewcited by 0
  15. 2024A comparison of RL-based and PID controllers for 6-DOF swimming robots: hybrid underwater object trackingcited by 0
  16. 2024A Survey on Applications of Reinforcement Learning in Spatial Resource Allocationcited by 0
  17. 2024On-policy Actor-Critic Reinforcement Learning for Multi-UAV Explorationcited by 0
  18. 2025RT-HCP: Dealing with Inference Delays and Sample Efficiency to Learn Directly on Robotic Platformscited by 0
  19. 2025A Survey of Freshness-Aware Wireless Networking with Reinforcement Learningcited by 0
  20. 2026Towards Near-Real-Time Telemetry-Aware Routing with Neural Routing Algorithmscited by 0
  21. 2026Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learningcited by 0
  22. 2026Temporal Self-Imitation Learningcited by 0
  23. 2026Think at 5 Hz, Act at 20 Hz: Asynchronous Fast-Slow Vision-Language-Action Inference for Closed-Loop Drivingcited by 0
  24. 2026RTNav: Towards Real-Time Zero-Shot Object Navigationcited by 0
  25. 2026GzDRL: Reproducible and Scalable Deep Reinforcement Learning with Gazebocited by 0
  26. 2026StreamDecisionBench: Evaluating Decisions in Force on Evolving Language Streamscited by 0