# i-Sim2Real: Reinforcement Learning of Robotic Policies in Tight Human-Robot Interaction Loops

arXiv: https://arxiv.org/abs/2207.06572 · HTML: https://mlsanity.com/paper/2207.06572/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2022.

## Built on by (most cited first, 30 of 32)

- [DribbleBot: Dynamic Legged Manipulation in the Wild](https://mlsanity.com/paper/2304.01159/) (2023)
- [GINA-3D: Learning to Generate Implicit Neural Assets in the Wild](https://mlsanity.com/paper/2304.02163/) (2023)
- [ODRL: A Benchmark for Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2410.20750/) (2024)
- [Creating a Dynamic Quadrupedal Robotic Goalkeeper with Reinforcement Learning](https://mlsanity.com/paper/2210.04435/) (2022)
- [SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention](https://mlsanity.com/paper/2312.01990/) (2023)
- [Learning coordinated badminton skills for legged manipulators](https://mlsanity.com/paper/2505.22974/) (2025)
- [Robotic Table Tennis: A Case Study into a High Speed Learning System](https://mlsanity.com/paper/2309.03315/) (2023)
- [SpinDOE: A ball spin estimation method for table tennis robot](https://mlsanity.com/paper/2303.03879/) (2023)
- [PGODE: Towards High-quality System Dynamics Modeling](https://mlsanity.com/paper/2311.06554/) (2023)
- [Off-Policy Evaluation for Human Feedback](https://mlsanity.com/paper/2310.07123/) (2023)
- [Global Convergence Guarantees for Federated Policy Gradient Methods with Adversaries](https://mlsanity.com/paper/2403.09940/) (2024)
- [Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policy](https://mlsanity.com/paper/2403.01734/) (2024)
- [Learning Diverse Robot Striking Motions with Diffusion Models and Kinematically Constrained Gradient Guidance](https://mlsanity.com/paper/2409.15528/) (2024)
- [LATTE-MV: Learning to Anticipate Table Tennis Hits from Monocular Videos](https://mlsanity.com/paper/2503.20936/) (2025)
- [Bi-Manual Block Assembly via Sim-to-Real Reinforcement Learning](https://mlsanity.com/paper/2303.14870/) (2023)
- [Multi-Camera Asynchronous Ball Localization and Trajectory Prediction with Factor Graphs and Human Poses](https://mlsanity.com/paper/2401.17185/) (2024)
- [NatSGD: A Dataset with Speech, Gestures, and Demonstrations for Robot Learning in Natural Human-Robot Interaction](https://mlsanity.com/paper/2403.02274/) (2024)
- [Advancing Investment Frontiers: Industry-grade Deep Reinforcement Learning for Portfolio Optimization](https://mlsanity.com/paper/2403.07916/) (2024)
- [Integrating Learning-Based Manipulation and Physics-Based Locomotion for Whole-Body Badminton Robot Control](https://mlsanity.com/paper/2504.17771/) (2025)
- [A Review On Safe Reinforcement Learning Using Lyapunov and Barrier Functions](https://mlsanity.com/paper/2508.09128/) (2025)
- [GenDOM: Generalizable One-shot Deformable Object Manipulation with Parameter-Aware Policy](https://mlsanity.com/paper/2309.09051/) (2023)
- [Robot Motion Planning using One-Step Diffusion with Noise-Optimized Approximate Motions](https://mlsanity.com/paper/2504.19652/) (2025)
- [RealMirror: A Comprehensive, Open-Source Vision-Language-Action Platform for Embodied AI](https://mlsanity.com/paper/2509.14687/) (2025)
- [GenORM: Generalizable One-shot Rope Manipulation with Parameter-Aware Policy](https://mlsanity.com/paper/2306.09872/) (2023)
- [vMFER: Von Mises-Fisher Experience Resampling Based on Uncertainty of Gradient Directions for Policy Improvement](https://mlsanity.com/paper/2405.08638/) (2024)
- [HARP: Human-Assisted Regrouping with Permutation Invariant Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2409.11741/) (2024)
- [Model-Free, Regret-Optimal Best Policy Identification in Online CMDPs](https://mlsanity.com/paper/2309.15395/) (2023)
- [SCPO: Safe Reinforcement Learning with Safety Critic Policy Optimization](https://mlsanity.com/paper/2311.00880/) (2023)
- [SuDA: Support-based Domain Adaptation for Sim2Real Motion Capture with Flexible Sensors](https://mlsanity.com/paper/2405.16152/) (2024)
- [Catching Spinning Table Tennis Balls in Simulation with End-to-End Curriculum Reinforcement Learning](https://mlsanity.com/paper/2503.01251/) (2025)

## Abstract (arXiv)

Sim-to-real transfer is a powerful paradigm for robotic reinforcement learning. The ability to train policies in simulation enables safe exploration and large-scale data collection quickly at low cost. However, prior works in sim-to-real transfer of robotic policies typically do not involve any human-robot interaction because accurately simulating human behavior is an open problem. In this work, our goal is to leverage the power of simulation to train robotic policies that are proficient at interacting with humans upon deployment. But there is a chicken and egg problem -- how to gather examples of a human interacting with a physical robot so as to model human behavior in simulation without already having a robot that is able to interact with a human? Our proposed method, Iterative-Sim-to-Real (i-S2R), attempts to address this. i-S2R bootstraps from a simple model of human behavior and alternates between training in simulation and deploying in the real world. In each iteration, both the human behavior model and the policy are refined. For all training we apply a new evolutionary search algorithm called Blackbox Gradient Sensing (BGS). We evaluate our method on a real world robotic table tennis setting, where the objective for the robot is to play cooperatively with a human player for as long as possible. Table tennis is a high-speed, dynamic task that requires the two players to react quickly to each other's moves, making for a challenging test bed for research on human-robot interaction. We present results on an industrial robotic arm that is able to cooperatively play table tennis with human players, achieving rallies of 22 successive hits on average and 150 at best. Further, for 80% of players, rally lengths are 70% to 175% longer compared to the sim-to-real plus fine-tuning (S2R+FT) baseline. For videos of our system in action, please see https://sites.google.com/view/is2r.