# Learning Visual Tracking and Reaching with Deep Reinforcement Learning on a UR10e Robotic Arm

arXiv: https://arxiv.org/abs/2308.14652 · HTML: https://mlsanity.com/paper/2308.14652/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 25 of 25)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [Soft Actor-Critic for Discrete Action Settings](https://mlsanity.com/paper/1910.07207/) (2019)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [Deep Variational Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1806.02426/) (2018)
- [The Limits and Potentials of Deep Learning for Robotics](https://mlsanity.com/paper/1804.06557/) (2018)
- [Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost](https://mlsanity.com/paper/1810.06045/) (2018)
- [Learning Navigation Behaviors End-to-End with AutoRL](https://mlsanity.com/paper/1809.10124/) (2018)
- [Data Efficient Reinforcement Learning for Legged Robots](https://mlsanity.com/paper/1907.03613/) (2019)
- [Learning Fast Adaptation with Meta Strategy Optimization](https://mlsanity.com/paper/1909.12995/) (2019)
- [Setting up a Reinforcement Learning Task with a Real-World Robot](https://mlsanity.com/paper/1803.07067/) (2018)
- [Deep Predictive Policy Training using Reinforcement Learning](https://mlsanity.com/paper/1703.00727/) (2017)
- [Variable Compliance Control for Robotic Peg-in-Hole Assembly: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2008.10224/) (2020)
- [Controlling Commercial Cooling Systems Using Reinforcement Learning](https://mlsanity.com/paper/2211.07357/) (2022)
- [Asynchronous Reinforcement Learning for Real-Time Control of Physical Robots](https://mlsanity.com/paper/2203.12759/) (2022)
- [Robotic Arm Control and Task Training through Deep Reinforcement Learning](https://mlsanity.com/paper/2005.02632/) (2020)

## Abstract (arXiv)

As technology progresses, industrial and scientific robots are increasingly being used in diverse settings. In many cases, however, programming the robot to perform such tasks is technically complex and costly. To maximize the utility of robots in industrial and scientific settings, they require the ability to quickly shift from one task to another. Reinforcement learning algorithms provide the potential to enable robots to learn optimal solutions to complete new tasks without directly reprogramming them. The current state-of-the-art in reinforcement learning, however, generally relies on fast simulations and parallelization to achieve optimal performance. These are often not possible in robotics applications. Thus, a significant amount of research is required to facilitate the efficient and safe, training and deployment of industrial and scientific reinforcement learning robots. This technical report outlines our initial research into the application of deep reinforcement learning on an industrial UR10e robot. The report describes the reinforcement learning environments created to facilitate policy learning with the UR10e, a robotic arm from Universal Robots, and presents our initial results in training deep Q-learning and proximal policy optimization agents on the developed reinforcement learning environments. Our results show that proximal policy optimization learns a better, more stable policy with less data than deep Q-learning. The corresponding code for this work is available at urlhttps://github.com/cbellinger27/bendRLreachertracker