# Real-Time Reinforcement Learning for Vision-Based Robotics Utilizing Local and Remote Computers

arXiv: https://arxiv.org/abs/2210.02317 · HTML: https://mlsanity.com/paper/2210.02317/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 5 Oct 2022.

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Learning to Walk via Deep Reinforcement Learning](https://mlsanity.com/paper/1812.11103/) (2018)
- [Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping](https://mlsanity.com/paper/1709.07857/) (2017)
- [Massively Parallel Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1507.04296/) (2015)
- [Benchmarking Reinforcement Learning Algorithms on Real-World Robots](https://mlsanity.com/paper/1809.07731/) (2018)
- [Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learning](https://mlsanity.com/paper/1901.03737/) (2019)
- [Real-Time Reinforcement Learning](https://mlsanity.com/paper/1911.04448/) (2019)
- [Learning Dexterous Manipulation for a Soft Robotic Hand from Human Demonstration](https://mlsanity.com/paper/1603.06348/) (2016)
- [Setting up a Reinforcement Learning Task with a Real-World Robot](https://mlsanity.com/paper/1803.07067/) (2018)
- [Asynchronous Reinforcement Learning for Real-Time Control of Physical Robots](https://mlsanity.com/paper/2203.12759/) (2022)

## Built on by (most cited first, 13 of 13)

- [Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning](https://mlsanity.com/paper/2404.00781/) (2024)
- [Streaming Deep Reinforcement Learning Finally Works](https://mlsanity.com/paper/2410.14606/) (2024)
- [Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers](https://mlsanity.com/paper/2411.15370/) (2024)
- [Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning](https://mlsanity.com/paper/2407.00324/) (2024)
- [MaDi: Learning to Mask Distractions for Generalization in Visual Deep Reinforcement Learning](https://mlsanity.com/paper/2312.15339/) (2023)
- [Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learning](https://mlsanity.com/paper/2406.03276/) (2024)
- [MARLIN: Multi-Agent Reinforcement Learning Guided by Language-Based Inter-Robot Negotiation](https://mlsanity.com/paper/2410.14383/) (2024)
- [Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path Planning](https://mlsanity.com/paper/2406.04920/) (2024)
- [Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrations](https://mlsanity.com/paper/2410.14957/) (2024)
- [Learning Without Time-Based Embodiment Resets in Soft-Actor Critic](https://mlsanity.com/paper/2512.06252/) (2025)
- [Incremental Residual Reinforcement Learning Toward Real-World Learning for Social Navigation](https://mlsanity.com/paper/2604.07945/) (2026)
- [Synchronous vs Asynchronous Reinforcement Learning in a Real World Robot](https://mlsanity.com/paper/2503.14554/) (2025)
- [Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/2603.08588/) (2026)

## Abstract (arXiv)

Real-time learning is crucial for robotic agents adapting to ever-changing, non-stationary environments. A common setup for a robotic agent is to have two different computers simultaneously: a resource-limited local computer tethered to the robot and a powerful remote computer connected wirelessly. Given such a setup, it is unclear to what extent the performance of a learning system can be affected by resource limitations and how to efficiently use the wirelessly connected powerful computer to compensate for any performance loss. In this paper, we implement a real-time learning system called the Remote-Local Distributed (ReLoD) system to distribute computations of two deep reinforcement learning (RL) algorithms, Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO), between a local and a remote computer. The performance of the system is evaluated on two vision-based control tasks developed using a robotic arm and a mobile robot. Our results show that SAC's performance degrades heavily on a resource-limited local computer. Strikingly, when all computations of the learning system are deployed on a remote workstation, SAC fails to compensate for the performance loss, indicating that, without careful consideration, using a powerful remote computer may not result in performance improvement. However, a carefully chosen distribution of computations of SAC consistently and substantially improves its performance on both tasks. On the other hand, the performance of PPO remains largely unaffected by the distribution of computations. In addition, when all computations happen solely on a powerful tethered computer, the performance of our system remains on par with an existing system that is well-tuned for using a single machine. ReLoD is the only publicly available system for real-time RL that applies to multiple robots for vision-based tasks.