# RoboFactory: Exploring Embodied Agent Collaboration with Compositional Constraints

arXiv: https://arxiv.org/abs/2503.16408 · HTML: https://mlsanity.com/paper/2503.16408/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 2 of its 2 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 2 of 2)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)](https://mlsanity.com/paper/2409.02920/) (2024)

## Built on by (most cited first, 30 of 31)

- [RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics](https://mlsanity.com/paper/2506.04308/) (2025)
- [TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Robotics](https://mlsanity.com/paper/2510.07181/) (2025)
- [A Survey of Interactive Generative Video](https://mlsanity.com/paper/2504.21853/) (2025)
- [MultiWorld: Scalable Multi-Agent Multi-View Video World Models](https://mlsanity.com/paper/2604.18564/) (2026)
- [VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learning](https://mlsanity.com/paper/2506.09049/) (2025)
- [RoboSafe: Safeguarding Embodied Agents via Executable Safety Logic](https://mlsanity.com/paper/2512.21220/) (2025)
- [HumanoidGen: Data Generation for Bimanual Dexterous Manipulation via LLM Reasoning](https://mlsanity.com/paper/2507.00833/) (2025)
- [CDP: Towards Robust Autoregressive Visuomotor Policy Learning via Causal Diffusion](https://mlsanity.com/paper/2506.14769/) (2025)
- [Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis](https://mlsanity.com/paper/2510.08759/) (2025)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [Towards Robust Evaluation of STEM Education: Leveraging MLLMs in Project-Based Learning](https://mlsanity.com/paper/2505.17050/) (2025)
- [LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge](https://mlsanity.com/paper/2511.01409/) (2025)
- [MA-VLA: Multi-Arm Vision-Language-Action Model for Collaboration and Compositional Generalization](https://mlsanity.com/paper/2608.25864/) (2026)
- [GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies](https://mlsanity.com/paper/2511.00998/) (2025)
- [Advances and Innovations in the Multi-Agent Robotic System (MARS) Challenge](https://mlsanity.com/paper/2601.18733/) (2026)
- [Ego to World: Collaborative Spatial Reasoning in Embodied Systems via Reinforcement Learning](https://mlsanity.com/paper/2603.14811/) (2026)
- [CoEnv: Driving Embodied Multi-Agent Collaboration via Compositional Environment](https://mlsanity.com/paper/2604.05484/) (2026)
- [When Replanning Becomes the Bottleneck: Budgeted Replanning for Embodied Agents](https://mlsanity.com/paper/2608.01428/) (2026)
- [GuideFetch: A Task Coordination Framework for Concurrent Navigation and Object Retrieval in Assistive Robot Dogs](https://mlsanity.com/paper/2608.18292/) (2026)
- [Referring-Aware Visuomotor Policy Learning for Closed-Loop Manipulation](https://mlsanity.com/paper/2604.05544/) (2026)
- [ComSim: Building Scalable Real-World Robot Data Generation via Compositional Simulation](https://mlsanity.com/paper/2604.11386/) (2026)
- [GuardAD: Safeguarding Autonomous Driving MLLMs via Markovian Safety Logic](https://mlsanity.com/paper/2605.10386/) (2026)
- [Episodic Memory Temporal Consistency for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2606.04492/) (2026)
- [ChronoPhyBench: Do MLLMs Truly Understand the World or Merely Exploit Language Priors?](https://mlsanity.com/paper/2606.07962/) (2026)
- [OccAnyScene: Towards Unified Indoor-Outdoor 3D Occupancy Prediction](https://mlsanity.com/paper/2608.08696/) (2026)
- [LTLDiff: Finite Linear Temporal Logic-Guided Data Generation and Diffusion Policies for Multi-agent Robotic Manipulation](https://mlsanity.com/paper/2609.11043/) (2026)
- [RoboFind: Multi-Agent Personalized Object Search for People Who Are Blind or Have Low Vision](https://mlsanity.com/paper/2609.20330/) (2026)
- [CARE: Experience-Guided Atomic Corrective Execution for Vision-Language-Action Policies](https://mlsanity.com/paper/2609.24118/) (2026)
- [CoHuB: A Simulation Benchmark for Multi-Humanoid Collaboration](https://mlsanity.com/paper/2609.34782/) (2026)
- [Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuning](https://mlsanity.com/paper/2609.36588/) (2026)

## Abstract (arXiv)

Designing effective embodied multi-agent systems is critical for solving complex real-world tasks across domains. Due to the complexity of multi-agent embodied systems, existing methods fail to automatically generate safe and efficient training data for such systems. To this end, we propose the concept of compositional constraints for embodied multi-agent systems, addressing the challenges arising from collaboration among embodied agents. We design various interfaces tailored to different types of constraints, enabling seamless interaction with the physical world. Leveraging compositional constraints and specifically designed interfaces, we develop an automated data collection framework for embodied multi-agent systems and introduce the first benchmark for embodied multi-agent manipulation, RoboFactory. Based on RoboFactory benchmark, we adapt and evaluate the method of imitation learning and analyzed its performance in different difficulty agent tasks. Furthermore, we explore the architectures and training strategies for multi-agent imitation learning, aiming to build safe and efficient embodied multi-agent systems.