# MBAPose: Mask and Bounding-Box Aware Pose Estimation of Surgical Instruments with Photorealistic Domain Randomization

arXiv: https://arxiv.org/abs/2103.08105 · HTML: https://mlsanity.com/paper/2103.08105/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 15 Mar 2021.

## Builds on (most cited first, 24 of 24)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [Strong-Weak Distribution Alignment for Adaptive Object Detection](https://mlsanity.com/paper/1812.04798/) (2018)
- [SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again](https://mlsanity.com/paper/1711.10006/) (2017)
- [A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay](https://mlsanity.com/paper/1803.09820/) (2018)
- [Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection](https://mlsanity.com/paper/1708.01642/) (2017)
- [Progressive Domain Adaptation for Object Detection](https://mlsanity.com/paper/1910.11319/) (2019)
- [DSSD : Deconvolutional Single Shot Detector](https://mlsanity.com/paper/1701.06659/) (2017)
- [Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing](https://mlsanity.com/paper/1810.08705/) (2018)
- [Segmentation-driven 6D Object Pose Estimation](https://mlsanity.com/paper/1812.02541/) (2018)
- [Deep Drone Racing: From Simulation to Reality with Domain Randomization](https://mlsanity.com/paper/1905.09727/) (2019)
- [Photorealistic Image Synthesis for Object Instance Detection](https://mlsanity.com/paper/1902.03334/) (2019)
- [Adapting Object Detectors with Conditional Domain Normalization](https://mlsanity.com/paper/2003.07071/) (2020)
- [Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering](https://mlsanity.com/paper/1907.04298/) (2019)
- [Camera-to-Robot Pose Estimation from a Single Image](https://mlsanity.com/paper/1911.09231/) (2019)
- [EfficientPose: An efficient, accurate and scalable end-to-end 6D multi object pose estimation approach](https://mlsanity.com/paper/2011.04307/) (2020)
- [Single-Shot Pose Estimation of Surgical Robot Instruments' Shafts from Monocular Endoscopic Images](https://mlsanity.com/paper/2003.01267/) (2020)
- [Dynamic Active Constraints for Surgical Robots using Vector Field Inequalities](https://mlsanity.com/paper/1804.11270/) (2018)
- [A Unified Framework for the Teleoperation of Surgical Robots in Constrained Workspaces](https://mlsanity.com/paper/1809.07907/) (2018)

## Built on by (most cited first, 3 of 3)

- [Online camera-pose-free stereo endoscopic tissue deformation recovery with tissue-invariant vision-biomechanics consistency](https://mlsanity.com/paper/2506.19388/) (2025)
- [MarUco: A Markerless 6D Pose Estimation Framework for Closed-Loop Control of Surgical Continuum Manipulators](https://mlsanity.com/paper/2602.16365/) (2026)
- [SurgCast: Action-Conditioned Future Skeletons for Controllable Surgical World Models](https://mlsanity.com/paper/2610.15645/) (2026)

## Abstract (arXiv)

Surgical robots are usually controlled using a priori models based on the robots' geometric parameters, which are calibrated before the surgical procedure. One of the challenges in using robots in real surgical settings is that those parameters can change over time, consequently deteriorating control accuracy. In this context, our group has been investigating online calibration strategies without added sensors. In one step toward that goal, we have developed an algorithm to estimate the pose of the instruments' shafts in endoscopic images. In this study, we build upon that earlier work and propose a new framework to more precisely estimate the pose of a rigid surgical instrument. Our strategy is based on a novel pose estimation model called MBAPose and the use of synthetic training data. Our experiments demonstrated an improvement of 21 % for translation error and 26 % for orientation error on synthetic test data with respect to our previous work. Results with real test data provide a baseline for further research.