# Image-to-Force Estimation for Soft Tissue Interaction in Robotic-Assisted Surgery Using Structured Light

arXiv: https://arxiv.org/abs/2501.08593 · HTML: https://mlsanity.com/paper/2501.08593/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 37 references that are in the map.
- First posted on arXiv 15 Jan 2025.

## Builds on (most cited first, 6 of 6)

- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch](https://mlsanity.com/paper/1805.11085/) (2018)
- [Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images](https://mlsanity.com/paper/1803.03435/) (2018)
- [Visual-Tactile Multimodality for Following Deformable Linear Objects Using Reinforcement Learning](https://mlsanity.com/paper/2204.00117/) (2022)
- [Toward Force Estimation in Robot-Assisted Surgery using Deep Learning with Vision and Robot State](https://mlsanity.com/paper/2011.02112/) (2020)
- [Force Sensing in Robot-assisted Keyhole Endoscopy: A Systematic Survey](https://mlsanity.com/paper/2103.11123/) (2021)

## Built on by (most cited first, 2 of 2)

- [Sim-to-Real Gentle Manipulation of Deformable and Fragile Objects with Stress-Guided Reinforcement Learning](https://mlsanity.com/paper/2510.25405/) (2025)
- [Latent Diffeomorphic Co-Design of End-Effectors for Deformable and Fragile Object Manipulation](https://mlsanity.com/paper/2602.17921/) (2026)

## Abstract (arXiv)

For Minimally Invasive Surgical (MIS) robots, accurate haptic interaction force feedback is essential for ensuring the safety of interacting with soft tissue. However, most existing MIS robotic systems cannot facilitate direct measurement of the interaction force with hardware sensors due to space limitations. This letter introduces an effective vision-based scheme that utilizes a One-Shot structured light projection with a designed pattern on soft tissue coupled with haptic information processing through a trained image-to-force neural network. The images captured from the endoscopic stereo camera are analyzed to reconstruct high-resolution 3D point clouds for soft tissue deformation. Based on this, a modified PointNet-based force estimation method is proposed, which excels in representing the complex mechanical properties of soft tissue. Numerical force interaction experiments are conducted on three silicon materials with different stiffness. The results validate the effectiveness of the proposed scheme.