# Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordination

arXiv: https://arxiv.org/abs/2610.02170 · HTML: https://mlsanity.com/paper/2610.02170/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Builds on (most cited first, 28 of 28)

- [Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration](https://mlsanity.com/paper/2010.09890/) (2020)
- [Learning Latent Representations to Influence Multi-Agent Interaction](https://mlsanity.com/paper/2011.06619/) (2020)
- [Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2011.09999/) (2020)
- [Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real](https://mlsanity.com/paper/1908.05224/) (2019)
- [Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior](https://mlsanity.com/paper/1805.08010/) (2018)
- [Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning](https://mlsanity.com/paper/1909.05477/) (2019)
- [NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants](https://mlsanity.com/paper/2301.05223/) (2023)
- [Benchmarking Constraint Inference in Inverse Reinforcement Learning](https://mlsanity.com/paper/2206.09670/) (2022)
- [Diffusion Co-Policy for Synergistic Human-Robot Collaborative Tasks](https://mlsanity.com/paper/2305.12171/) (2023)
- [Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge](https://mlsanity.com/paper/2411.01796/) (2024)
- [Smart Help: Strategic Opponent Modeling for Proactive and Adaptive Robot Assistance in Households](https://mlsanity.com/paper/2404.09001/) (2024)
- [Learning Parametric Constraints in High Dimensions from Demonstrations](https://mlsanity.com/paper/1910.03477/) (2019)
- [Latent Theory of Mind: A Decentralized Diffusion Architecture for Cooperative Manipulation](https://mlsanity.com/paper/2505.09144/) (2025)
- [Getting to Know One Another: Calibrating Intent, Capabilities and Trust for Human-Robot Collaboration](https://mlsanity.com/paper/2008.00699/) (2020)
- [MIMIC-D: Multi-modal Imitation for MultI-agent Coordination with Decentralized Diffusion Policies](https://mlsanity.com/paper/2509.14159/) (2025)
- [CLAS: Coordinating Multi-Robot Manipulation with Central Latent Action Spaces](https://mlsanity.com/paper/2211.15824/) (2022)
- [CHORUS: Decentralized Multi-Embodiment Collaboration with One VLA Policy](https://mlsanity.com/paper/2606.12352/) (2026)
- [Learning Decentralized Multi-Biped Control for Payload Transport](https://mlsanity.com/paper/2406.17279/) (2024)
- [GRACE: Generalizing Robot-Assisted Caregiving with User Functionality Embeddings](https://mlsanity.com/paper/2501.17855/) (2025)
- [Multi-Quadruped Cooperative Object Transport: Learning Decentralized Pinch-Lift-Move](https://mlsanity.com/paper/2509.14342/) (2025)
- [Deep reinforcement learning of event-triggered communication and consensus-based control for distributed cooperative transport](https://mlsanity.com/paper/2212.01958/) (2022)
- [Implicit Communication in Human-Robot Collaborative Transport](https://mlsanity.com/paper/2502.03346/) (2025)
- [CollaBot: Vision-Language Guided Simultaneous Collaborative Manipulation](https://mlsanity.com/paper/2508.03526/) (2025)
- [Safe and Interpretable Multimodal Path Planning for Multi-Agent Cooperation](https://mlsanity.com/paper/2602.19304/) (2026)
- [Decentralized Ability-Aware Adaptive Control for Multi-robot Collaborative Manipulation](https://mlsanity.com/paper/2102.03689/) (2021)
- [MR.CAP: Multi-Robot Joint Control and Planning for Object Transport](https://mlsanity.com/paper/2401.11634/) (2024)
- [Constraint Learning in Multi-Agent Dynamic Games from Demonstrations of Local Nash Interactions](https://mlsanity.com/paper/2508.19945/) (2025)
- [Partner Capability Estimation for Task-Agnostic Adaptation in Ad-Hoc Teamwork](https://mlsanity.com/paper/2607.27177/) (2026)

## Abstract (arXiv)

Robots operating in the physical world will increasingly need to coordinate with other robots, particularly in manipulation tasks where an object may be too large or heavy for a single robot to carry alone. Physical limitations caused by hardware degradation or actuator faults can restrict the actions a robot can reliably execute, yet these limitations may be unknown to its partner. We study whether a helper can infer a robot partner's physical constraints from observing it coordinate with another robot, then use the inferred capability to coordinate with the same partner on a new task. This is difficult because a demonstration shows what the constrained robot did, but not what it could have done. In physically coupled tasks, the other robot may also compensate for its limitations, making those limitations difficult to identify from the constrained robot's behavior alone. Our key insight is that these constraints shape the joint behavior of the team, making the actions of both robots informative about the constrained partner's capability. We introduce Watch, Infer, Coordinate, a benchmark spanning three physically coupled manipulation settings, together with an inference approach that scores candidate constraints using observed joint behavior. Across all three settings, our method substantially improves constraint inference and zero-shot coordination, approaching an oracle with access to the true constraints.