# 3D Simulation for Robot Arm Control with Deep Q-Learning

arXiv: https://arxiv.org/abs/1609.03759 · HTML: https://mlsanity.com/paper/1609.03759/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2016.

## Built on by (most cited first, 21 of 21)

- [Real-World Robot Learning with Masked Visual Pre-training](https://mlsanity.com/paper/2210.03109/) (2022)
- [Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation](https://mlsanity.com/paper/2106.12534/) (2021)
- [Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions](https://mlsanity.com/paper/2203.15103/) (2022)
- [Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation](https://mlsanity.com/paper/2105.14829/) (2021)
- [Task-Embedded Control Networks for Few-Shot Imitation Learning](https://mlsanity.com/paper/1810.03237/) (2018)
- [VeLO: Training Versatile Learned Optimizers by Scaling Up](https://mlsanity.com/paper/2211.09760/) (2022)
- [Generative Image as Action Models](https://mlsanity.com/paper/2407.07875/) (2024)
- [Analysis of Randomization Effects on Sim2Real Transfer in Reinforcement Learning for Robotic Manipulation Tasks](https://mlsanity.com/paper/2206.06282/) (2022)
- [Coarse-to-Fine Q-attention with Learned Path Ranking](https://mlsanity.com/paper/2204.01571/) (2022)
- [Adversarial Counterfactual Environment Model Learning](https://mlsanity.com/paper/2206.04890/) (2022)
- [Coarse-to-fine Q-attention with Tree Expansion](https://mlsanity.com/paper/2204.12471/) (2022)
- [Continual Domain Randomization](https://mlsanity.com/paper/2403.12193/) (2024)
- [DiAReL: Reinforcement Learning with Disturbance Awareness for Robust Sim2Real Policy Transfer in Robot Control](https://mlsanity.com/paper/2306.09010/) (2023)
- [Validate on Sim, Detect on Real -- Model Selection for Domain Randomization](https://mlsanity.com/paper/2111.00765/) (2021)
- [VMAgent: Scheduling Simulator for Reinforcement Learning](https://mlsanity.com/paper/2112.04785/) (2021)
- [Document Domain Randomization for Deep Learning Document Layout Extraction](https://mlsanity.com/paper/2105.14931/) (2021)
- [Neural Task Success Classifiers for Robotic Manipulation from Few Real Demonstrations](https://mlsanity.com/paper/2107.00722/) (2021)
- [Robust Meta-Reinforcement Learning with Curriculum-Based Task Sampling](https://mlsanity.com/paper/2203.16801/) (2022)
- [Enhancing team performance with transfer-learning during real-world human-robot collaboration](https://mlsanity.com/paper/2211.13070/) (2022)
- [Rapid Training Data Creation by Synthesizing Medical Images for Classification and Localization](https://mlsanity.com/paper/2308.04687/) (2023)
- [Dynamic Q-planning for Online UAV Path Planning in Unknown and Complex Environments](https://mlsanity.com/paper/2402.06297/) (2024)

## Abstract (arXiv)

Recent trends in robot arm control have seen a shift towards end-to-end solutions, using deep reinforcement learning to learn a controller directly from raw sensor data, rather than relying on a hand-crafted, modular pipeline. However, the high dimensionality of the state space often means that it is impractical to generate sufficient training data with real-world experiments. As an alternative solution, we propose to learn a robot controller in simulation, with the potential of then transferring this to a real robot. Building upon the recent success of deep Q-networks, we present an approach which uses 3D simulations to train a 7-DOF robotic arm in a control task without any prior knowledge. The controller accepts images of the environment as its only input, and outputs motor actions for the task of locating and grasping a cube, over a range of initial configurations. To encourage efficient learning, a structured reward function is designed with intermediate rewards. We also present preliminary results in direct transfer of policies over to a real robot, without any further training.