# Bi-Manual Joint Camera Calibration and Scene Representation

arXiv: https://arxiv.org/abs/2505.24819 · HTML: https://mlsanity.com/paper/2505.24819/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 51 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Foundation Models in Robotics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2312.07843/) (2023)
- [Cameras as Rays: Pose Estimation via Ray Diffusion](https://mlsanity.com/paper/2402.14817/) (2024)
- [Grasp Pose Detection in Point Clouds](https://mlsanity.com/paper/1706.09911/) (2017)
- [Deep Regression on Manifolds: A 3D Rotation Case Study](https://mlsanity.com/paper/2103.16317/) (2021)
- [DarkGS: Learning Neural Illumination and 3D Gaussians Relighting for Robotic Exploration in the Dark](https://mlsanity.com/paper/2403.10814/) (2024)
- [Instructing Robots by Sketching: Learning from Demonstration via Probabilistic Diagrammatic Teaching](https://mlsanity.com/paper/2309.03835/) (2023)
- [Learning Eye-in-Hand Camera Calibration from a Single Image](https://mlsanity.com/paper/2111.01245/) (2021)
- [3D Foundation Models Enable Simultaneous Geometry and Pose Estimation of Grasped Objects](https://mlsanity.com/paper/2407.10331/) (2024)
- [GraphSeg: Segmented 3D Representations via Graph Edge Addition and Contraction](https://mlsanity.com/paper/2504.03129/) (2025)
- [Simultaneously Calibration of Multi Hand-Eye Robot System Based on Graph](https://mlsanity.com/paper/2305.02518/) (2023)

## Built on by (most cited first, 2 of 2)

- [PATCH: Action-Chunk-Conditioned Latent Patch Innovation Monitoring for Robot Manipulation](https://mlsanity.com/paper/2606.16690/) (2026)
- [MAVP: Map-Aware Visuomotor Policies for Mobile Manipulation](https://mlsanity.com/paper/2609.26378/) (2026)

## Abstract (arXiv)

Robot manipulation, especially bimanual manipulation, often requires setting up multiple cameras on multiple robot manipulators. Before robot manipulators can generate motion or even build representations of their environments, the cameras rigidly mounted to the robot need to be calibrated. Camera calibration is a cumbersome process involving collecting a set of images, with each capturing a pre-determined marker. In this work, we introduce the Bi-Manual Joint Calibration and Representation Framework (Bi-JCR). Bi-JCR enables multiple robot manipulators, each with cameras mounted, to circumvent taking images of calibration markers. By leveraging 3D foundation models for dense, marker-free multi-view correspondence, Bi-JCR jointly estimates: (i) the extrinsic transformation from each camera to its end-effector, (ii) the inter-arm relative poses between manipulators, and (iii) a unified, scale-consistent 3D representation of the shared workspace, all from the same captured RGB image sets. The representation, jointly constructed from images captured by cameras on both manipulators, lives in a common coordinate frame and supports collision checking and semantic segmentation to facilitate downstream bimanual coordination tasks. We empirically evaluate the robustness of Bi-JCR on a variety of tabletop environments, and demonstrate its applicability on a variety of downstream tasks.