# Learning Multi-Arm Manipulation Through Collaborative Teleoperation

arXiv: https://arxiv.org/abs/2012.06738 · HTML: https://mlsanity.com/paper/2012.06738/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 12 Dec 2020.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Built on by (most cited first, 30 of 32)

- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System](https://mlsanity.com/paper/2307.04577/) (2023)
- [DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning](https://mlsanity.com/paper/2410.24185/) (2024)
- [Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation](https://mlsanity.com/paper/2109.13841/) (2021)
- [Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning](https://mlsanity.com/paper/2407.03162/) (2024)
- [LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discovery](https://mlsanity.com/paper/2311.02058/) (2023)
- [Imitating Task and Motion Planning with Visuomotor Transformers](https://mlsanity.com/paper/2305.16309/) (2023)
- [Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation](https://mlsanity.com/paper/2112.05251/) (2021)
- [Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation](https://mlsanity.com/paper/2309.01952/) (2023)
- [SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment](https://mlsanity.com/paper/2410.18907/) (2024)
- [Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/2503.24361/) (2025)
- [TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulation](https://mlsanity.com/paper/2403.07869/) (2024)
- [Bi-ACT: Bilateral Control-Based Imitation Learning via Action Chunking with Transformer](https://mlsanity.com/paper/2401.17698/) (2024)
- [TidyBot++: An Open-Source Holonomic Mobile Manipulator for Robot Learning](https://mlsanity.com/paper/2412.10447/) (2024)
- [ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection](https://mlsanity.com/paper/2405.03666/) (2024)
- [Interactive Imitation Learning of Bimanual Movement Primitives](https://mlsanity.com/paper/2210.16220/) (2022)
- [Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration](https://mlsanity.com/paper/2108.06038/) (2021)
- [VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions](https://mlsanity.com/paper/2410.20927/) (2024)
- [DABI: Evaluation of Data Augmentation Methods Using Downsampling in Bilateral Control-Based Imitation Learning with Images](https://mlsanity.com/paper/2410.04370/) (2024)
- [Benchmarking Vision, Language, & Action Models in Procedurally Generated, Open Ended Action Environments](https://mlsanity.com/paper/2505.05540/) (2025)
- [CHORUS: Decentralized Multi-Embodiment Collaboration with One VLA Policy](https://mlsanity.com/paper/2606.12352/) (2026)
- [DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation](https://mlsanity.com/paper/2106.05907/) (2021)
- [Collaborative Navigation and Manipulation of a Cable-towed Load by Multiple Quadrupedal Robots](https://mlsanity.com/paper/2206.14424/) (2022)
- [IRIS: An Immersive Robot Interaction System](https://mlsanity.com/paper/2502.03297/) (2025)
- [Automated Hybrid Reward Scheduling via Large Language Models for Robotic Skill Learning](https://mlsanity.com/paper/2505.02483/) (2025)
- [Single RGB-D Camera Teleoperation for General Robotic Manipulation](https://mlsanity.com/paper/2106.14396/) (2021)
- [Collaborative motion planning for multi-manipulator systems through Reinforcement Learning and Dynamic Movement Primitives](https://mlsanity.com/paper/2410.00757/) (2024)
- [See Selectively, Act Adaptively: Dual-Level Structural Decomposition for Bimanual Robot Manipulation](https://mlsanity.com/paper/2606.13279/) (2026)
- [AR as an Evaluation Playground: Bridging Metrics and Visual Perception of Computer Vision Models](https://mlsanity.com/paper/2508.04102/) (2025)

## Abstract (arXiv)

Imitation Learning (IL) is a powerful paradigm to teach robots to perform manipulation tasks by allowing them to learn from human demonstrations collected via teleoperation, but has mostly been limited to single-arm manipulation. However, many real-world tasks require multiple arms, such as lifting a heavy object or assembling a desk. Unfortunately, applying IL to multi-arm manipulation tasks has been challenging -- asking a human to control more than one robotic arm can impose significant cognitive burden and is often only possible for a maximum of two robot arms. To address these challenges, we present Multi-Arm RoboTurk (MART), a multi-user data collection platform that allows multiple remote users to simultaneously teleoperate a set of robotic arms and collect demonstrations for multi-arm tasks. Using MART, we collected demonstrations for five novel two and three-arm tasks from several geographically separated users. From our data we arrived at a critical insight: most multi-arm tasks do not require global coordination throughout its full duration, but only during specific moments. We show that learning from such data consequently presents challenges for centralized agents that directly attempt to model all robot actions simultaneously, and perform a comprehensive study of different policy architectures with varying levels of centralization on our tasks. Finally, we propose and evaluate a base-residual policy framework that allows trained policies to better adapt to the mixed coordination setting common in multi-arm manipulation, and show that a centralized policy augmented with a decentralized residual model outperforms all other models on our set of benchmark tasks. Additional results and videos at https://roboturk.stanford.edu/multiarm .