# Polybot: Training One Policy Across Robots While Embracing Variability

arXiv: https://arxiv.org/abs/2307.03719 · HTML: https://mlsanity.com/paper/2307.03719/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 70 references that are in the map.
- First posted on arXiv 7 Jul 2023.

## Builds on (most cited first, 30 of 53)

- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [RoboNet: Large-Scale Multi-Robot Learning](https://mlsanity.com/paper/1910.11215/) (2019)
- [Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets](https://mlsanity.com/paper/2109.13396/) (2021)
- [Real-World Robot Learning with Masked Visual Pre-training](https://mlsanity.com/paper/2210.03109/) (2022)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience](https://mlsanity.com/paper/1810.05687/) (2018)
- [RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation](https://mlsanity.com/paper/1811.02790/) (2018)
- [Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning](https://mlsanity.com/paper/1511.06342/) (2015)
- [Multi-Task Reinforcement Learning with Context-based Representations](https://mlsanity.com/paper/2102.06177/) (2021)
- [GNM: A General Navigation Model to Drive Any Robot](https://mlsanity.com/paper/2210.03370/) (2022)
- [Language-Driven Representation Learning for Robotics](https://mlsanity.com/paper/2302.12766/) (2023)
- [Human-to-Robot Imitation in the Wild](https://mlsanity.com/paper/2207.09450/) (2022)
- [Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping](https://mlsanity.com/paper/1709.07857/) (2017)
- [Preparing for the Unknown: Learning a Universal Policy with Online System Identification](https://mlsanity.com/paper/1702.02453/) (2017)
- [Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer](https://mlsanity.com/paper/1609.07088/) (2016)

## Built on by (most cited first, 28 of 28)

- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation](https://mlsanity.com/paper/2410.07864/) (2024)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [EgoMimic: Scaling Imitation Learning via Egocentric Video](https://mlsanity.com/paper/2410.24221/) (2024)
- [Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers](https://mlsanity.com/paper/2409.20537/) (2024)
- [Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation](https://mlsanity.com/paper/2408.11812/) (2024)
- [Humanoid Policy ~ Human Policy](https://mlsanity.com/paper/2503.13441/) (2025)
- [Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation](https://mlsanity.com/paper/2410.08001/) (2024)
- [FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policies](https://mlsanity.com/paper/2509.04996/) (2025)
- [Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learning](https://mlsanity.com/paper/2408.14037/) (2024)
- [Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners](https://mlsanity.com/paper/2505.23150/) (2025)
- [A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulation](https://mlsanity.com/paper/2602.01067/) (2026)
- [A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agents](https://mlsanity.com/paper/2402.04580/) (2024)
- [REGENT: A Retrieval-Augmented Generalist Agent That Can Act In-Context in New Environments](https://mlsanity.com/paper/2412.04759/) (2024)
- [OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learning](https://mlsanity.com/paper/2512.13100/) (2025)
- [xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editing](https://mlsanity.com/paper/2409.08687/) (2024)
- [Tool-as-Interface: Learning Robot Policies from Observing Human Tool Use](https://mlsanity.com/paper/2504.04612/) (2025)
- [Multi-Loco: Unifying Multi-Embodiment Legged Locomotion via Reinforcement Learning Augmented Diffusion](https://mlsanity.com/paper/2506.11470/) (2025)
- [AnyBody: A Benchmark Suite for Cross-Embodiment Manipulation](https://mlsanity.com/paper/2505.14986/) (2025)
- [Invariance Co-training for Robot Visual Generalization](https://mlsanity.com/paper/2512.05230/) (2025)
- [Reinforcement Learning from Wild Animal Videos](https://mlsanity.com/paper/2412.04273/) (2024)
- [Inference-stage Adaptation-projection Strategy Adapts Diffusion Policy to Cross-manipulators Scenarios](https://mlsanity.com/paper/2509.11621/) (2025)
- [Towards Natural Language-Driven Assembly Using Foundation Models](https://mlsanity.com/paper/2406.16093/) (2024)
- [Cross-Embodiment Transfer via Behavior-Aligned Representations](https://mlsanity.com/paper/2607.27549/) (2026)
- [The Embodiment Gap in Robot Foundation Models](https://mlsanity.com/paper/2608.18433/) (2026)
- [Cross-Embodied Affordance Transfer through Learning Affordance Equivalences](https://mlsanity.com/paper/2404.15648/) (2024)
- [Enhancing Linguistic Generalization of VLA: Fine-Tuning OpenVLA via Synthetic Instruction Augmentation](https://mlsanity.com/paper/2603.16044/) (2026)

## Abstract (arXiv)

Reusing large datasets is crucial to scale vision-based robotic manipulators to everyday scenarios due to the high cost of collecting robotic datasets. However, robotic platforms possess varying control schemes, camera viewpoints, kinematic configurations, and end-effector morphologies, posing significant challenges when transferring manipulation skills from one platform to another. To tackle this problem, we propose a set of key design decisions to train a single policy for deployment on multiple robotic platforms. Our framework first aligns the observation and action spaces of our policy across embodiments via utilizing wrist cameras and a unified, but modular codebase. To bridge the remaining domain shift, we align our policy's internal representations across embodiments through contrastive learning. We evaluate our method on a dataset collected over 60 hours spanning 6 tasks and 3 robots with varying joint configurations and sizes: the WidowX 250S, the Franka Emika Panda, and the Sawyer. Our results demonstrate significant improvements in success rate and sample efficiency for our policy when using new task data collected on a different robot, validating our proposed design decisions. More details and videos can be found on our anonymized project website: https://sites.google.com/view/polybot-multirobot