# Sim-and-Real Reinforcement Learning for Manipulation: A Consensus-based Approach

arXiv: https://arxiv.org/abs/2302.13423 · HTML: https://mlsanity.com/paper/2302.13423/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 26 Feb 2023.

## Builds on (most cited first, 18 of 18)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience](https://mlsanity.com/paper/1810.05687/) (2018)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning](https://mlsanity.com/paper/1803.09956/) (2018)
- [Deep Reinforcement Learning for High Precision Assembly Tasks](https://mlsanity.com/paper/1708.04033/) (2017)
- [Dot-to-Dot: Explainable Hierarchical Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/1904.06703/) (2019)
- [Learning a visuomotor controller for real world robotic grasping using simulated depth images](https://mlsanity.com/paper/1706.04652/) (2017)
- [Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight](https://mlsanity.com/paper/1902.03701/) (2019)
- [A Deep Learning Approach to Grasping the Invisible](https://mlsanity.com/paper/1909.04840/) (2019)
- [Reinforcement Learned Distributed Multi-Robot Navigation with Reciprocal Velocity Obstacle Shaped Rewards](https://mlsanity.com/paper/2203.10229/) (2022)
- [Zero-shot sim-to-real transfer of tactile control policies for aggressive swing-up manipulation](https://mlsanity.com/paper/2101.02680/) (2021)
- [SafeAPT: Safe Simulation-to-Real Robot Learning using Diverse Policies Learned in Simulation](https://mlsanity.com/paper/2201.13248/) (2022)
- [Sim and Real: Better Together](https://mlsanity.com/paper/2110.00445/) (2021)
- [MetaGrasp: Data Efficient Grasping by Affordance Interpreter Network](https://mlsanity.com/paper/1902.06554/) (2019)

## Built on by (most cited first, 2 of 2)

- [Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applications](https://mlsanity.com/paper/2408.10215/) (2024)
- [Sim-to-Real Deep Reinforcement Learning with Manipulators for Pick-and-place](https://mlsanity.com/paper/2309.09247/) (2023)

## Abstract (arXiv)

Sim-and-real training is a promising alternative to sim-to-real training for robot manipulations. However, the current sim-and-real training is neither efficient, i.e., slow convergence to the optimal policy, nor effective, i.e., sizeable real-world robot data. Given limited time and hardware budgets, the performance of sim-and-real training is not satisfactory. In this paper, we propose a Consensus-based Sim-And-Real deep reinforcement learning algorithm (CSAR) for manipulator pick-and-place tasks, which shows comparable performance in both sim-and-real worlds. In this algorithm, we train the agents in simulators and the real world to get the optimal policies for both sim-and-real worlds. We found two interesting phenomenons: (1) Best policy in simulation is not the best for sim-and-real training. (2) The more simulation agents, the better sim-and-real training. The experimental video is available at: https://youtu.be/mcHJtNIsTEQ.