# Kalib: Easy Hand-Eye Calibration with Reference Point Tracking

arXiv: https://arxiv.org/abs/2408.10562 · HTML: https://mlsanity.com/paper/2408.10562/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 39 references that are in the map.
- First posted on arXiv 20 Aug 2024.

## Builds on (most cited first, 15 of 15)

- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement](https://mlsanity.com/paper/2306.08637/) (2023)
- [SpatialTracker: Tracking Any 2D Pixels in 3D Space](https://mlsanity.com/paper/2404.04319/) (2024)
- [DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Video](https://mlsanity.com/paper/2403.14548/) (2024)
- [Camera-to-Robot Pose Estimation from a Single Image](https://mlsanity.com/paper/1911.09231/) (2019)
- [Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer](https://mlsanity.com/paper/2302.14332/) (2023)
- [EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration](https://mlsanity.com/paper/2305.01191/) (2023)
- [Learning Eye-in-Hand Camera Calibration from a Single Image](https://mlsanity.com/paper/2111.01245/) (2021)
- [Hand-Eye Calibration](https://mlsanity.com/paper/2311.12655/) (2023)
- [Learning Markerless Robot-Depth Camera Calibration and End-Effector Pose Estimation](https://mlsanity.com/paper/2212.07567/) (2022)
- [Robot Hand-Eye Calibration using Structure-from-Motion](https://mlsanity.com/paper/2311.11808/) (2023)

## Built on by (most cited first, 1 of 1)

- [SurgCalib: Gaussian Splatting-Based Hand-Eye Calibration for Robot-Assisted Minimally Invasive Surgery](https://mlsanity.com/paper/2603.08983/) (2026)

## Abstract (arXiv)

Hand-eye calibration aims to estimate the transformation between a camera and a robot. Traditional methods rely on fiducial markers, which require considerable manual effort and precise setup. Recent advances in deep learning have introduced markerless techniques but come with more prerequisites, such as retraining networks for each robot, and accessing accurate mesh models for data generation. In this paper, we propose Kalib, an automatic and easy-to-setup hand-eye calibration method that leverages the generalizability of visual foundation models to overcome these challenges. It features only two basic prerequisites, the robot's kinematic chain and a predefined reference point on the robot. During calibration, the reference point is tracked in the camera space. Its corresponding 3D coordinates in the robot coordinate can be inferred by forward kinematics. Then, a PnP solver directly estimates the transformation between the camera and the robot without training new networks or accessing mesh models. Evaluations in simulated and real-world benchmarks show that Kalib achieves good accuracy with a lower manual workload compared with recent baseline methods. We also demonstrate its application in multiple real-world settings with various robot arms and grippers. Kalib's user-friendly design and minimal setup requirements make it a possible solution for continuous operation in unstructured environments.