# GISR: Geometric Initialization and Silhouette-based Refinement for Single-View Robot Pose and Configuration Estimation

arXiv: https://arxiv.org/abs/2405.04890 · HTML: https://mlsanity.com/paper/2405.04890/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 8 May 2024.

## Builds on (most cited first, 22 of 22)

- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects](https://mlsanity.com/paper/1809.10790/) (2018)
- [Real-Time Seamless Single Shot 6D Object Pose Prediction](https://mlsanity.com/paper/1711.08848/) (2017)
- [6-DoF Object Pose from Semantic Keypoints](https://mlsanity.com/paper/1703.04670/) (2017)
- [RePOSE: Fast 6D Object Pose Refinement via Deep Texture Rendering](https://mlsanity.com/paper/2104.00633/) (2021)
- [Deep Model-Based 6D Pose Refinement in RGB](https://mlsanity.com/paper/1810.03065/) (2018)
- [Camera-to-Robot Pose Estimation from a Single Image](https://mlsanity.com/paper/1911.09231/) (2019)
- [Single-view robot pose and joint angle estimation via render & compare](https://mlsanity.com/paper/2104.09359/) (2021)
- [Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer](https://mlsanity.com/paper/2302.14332/) (2023)
- [CRAVES: Controlling Robotic Arm with a Vision-based Economic System](https://mlsanity.com/paper/1812.00725/) (2018)
- [Pose Estimation for Robot Manipulators via Keypoint Optimization and Sim-to-Real Transfer](https://mlsanity.com/paper/2010.08054/) (2020)
- [Image-based Pose Estimation and Shape Reconstruction for Robot Manipulators and Soft, Continuum Robots via Differentiable Rendering](https://mlsanity.com/paper/2302.14039/) (2023)
- [Indirect Object-to-Robot Pose Estimation from an External Monocular RGB Camera](https://mlsanity.com/paper/2008.11822/) (2020)
- [Riemannian Optimization for Distance-Geometric Inverse Kinematics](https://mlsanity.com/paper/2108.13720/) (2021)
- [Towards Robotic Eye Surgery: Marker-free, Online Hand-eye Calibration using Optical Coherence Tomography Images](https://mlsanity.com/paper/1808.05805/) (2018)
- [A Distance-Geometric Method for Recovering Robot Joint Angles From an RGB Image](https://mlsanity.com/paper/2301.02051/) (2023)
- [Convex Iteration for Distance-Geometric Inverse Kinematics](https://mlsanity.com/paper/2109.03374/) (2021)

## Built on by (most cited first, 1 of 1)

- [ToolTipNet: A Segmentation-Driven Deep Learning Baseline for Surgical Instrument Tip Detection](https://mlsanity.com/paper/2504.09700/) (2025)

## Abstract (arXiv)

In autonomous robotics, measurement of the robot's internal state and perception of its environment, including interaction with other agents such as collaborative robots, are essential. Estimating the pose of the robot arm from a single view has the potential to replace classical eye-to-hand calibration approaches and is particularly attractive for online estimation and dynamic environments. In addition to its pose, recovering the robot configuration provides a complete spatial understanding of the observed robot that can be used to anticipate the actions of other agents in advanced robotics use cases. Furthermore, this additional redundancy enables the planning and execution of recovery protocols in case of sensor failures or external disturbances. We introduce GISR - a deep configuration and robot-to-camera pose estimation method that prioritizes execution in real-time. GISR consists of two modules: (i) a geometric initialization module that efficiently computes an approximate robot pose and configuration, and (ii) a deep iterative silhouette-based refinement module that arrives at a final solution in just a few iterations. We evaluate GISR on publicly available data and show that it outperforms existing methods of the same class in terms of both speed and accuracy, and can compete with approaches that rely on ground-truth proprioception and recover only the pose.