mlsanityData to 7 Oct 2026

Real-Time Reinforcement Learning for Vision-Based Robotics Utilizing Local and Remote Computers

arXiv 2210.02317 ↗First posted 5 Oct 2022cs.RO cs.AI cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Real-time learning is crucial for robotic agents adapting to ever-changing, non-stationary environments. A common setup for a robotic agent is to have two different computers simultaneously: a resource-limited local computer tethered to the robot and a powerful remote computer connected wirelessly. Given such a setup, it is unclear to what extent the performance of a learning system can be affected by resource limitations and how to efficiently use the wirelessly connected powerful computer to compensate for any performance loss. In this paper, we implement a real-time learning system called the Remote-Local Distributed (ReLoD) system to distribute computations of two deep reinforcement learning (RL) algorithms, Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO), between a local and a remote computer. The performance of the system is evaluated on two vision-based control tasks developed using a robotic arm and a mobile robot. Our results show that SAC's performance degrades heavily on a resource-limited local computer. Strikingly, when all computations of the learning system are deployed on a remote workstation, SAC fails to compensate for the performance loss, indicating that, without careful consideration, using a powerful remote computer may not result in performance improvement. However, a carefully chosen distribution of computations of SAC consistently and substantially improves its performance on both tasks. On the other hand, the performance of PPO remains largely unaffected by the distribution of computations. In addition, when all computations happen solely on a powerful tethered computer, the performance of our system remains on par with an existing system that is well-tuned for using a single machine. ReLoD is the only publicly available system for real-time RL that applies to multiple robots for vision-based tasks.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 15 of its 27 references that are in the map. First posted on arXiv 5 Oct 2022.

Builds on

15 of 27 references.

References · 15

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  5. 2020Reinforcement Learning with Augmented Datacited by 452
  6. 2018Maximum a Posteriori Policy Optimisationcited by 318
  7. 2018Learning to Walk via Deep Reinforcement Learningcited by 153
  8. 2017Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Graspingcited by 138
  9. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 80
  10. 2018Benchmarking Reinforcement Learning Algorithms on Real-World Robotscited by 64
  11. 2019Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learningcited by 40
  12. 2019Real-Time Reinforcement Learningcited by 38
3 more
  1. 2016Learning Dexterous Manipulation for a Soft Robotic Hand from Human Demonstrationcited by 35
  2. 2018Setting up a Reinforcement Learning Task with a Real-World Robotcited by 32
  3. 2022Asynchronous Reinforcement Learning for Real-Time Control of Physical Robotscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 2; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 49
  2. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  3. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 16
  4. 2024Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learningcited by 12
  5. 2023MaDi: Learning to Mask Distractions for Generalization in Visual Deep Reinforcement Learningcited by 9
  6. 2024Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learningcited by 5
  7. 2024MARLIN: Multi-Agent Reinforcement Learning Guided by Language-Based Inter-Robot Negotiationcited by 5
  8. 2024Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path Planningcited by 2
  9. 2024Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrationscited by 2
  10. 2025Learning Without Time-Based Embodiment Resets in Soft-Actor Criticcited by 1
  11. 2025Synchronous vs Asynchronous Reinforcement Learning in a Real World Robotcited by 0
  12. 2026Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Controlcited by 0
1 more
  1. 2026Incremental Residual Reinforcement Learning Toward Real-World Learning for Social Navigationcited by 0