# Characterization of Real-time Haptic Feedback from Multimodal Neural Network-based Force Estimates during Teleoperation

arXiv: https://arxiv.org/abs/2109.11488 · HTML: https://mlsanity.com/paper/2109.11488/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2021.

## Built on by (most cited first, 1 of 1)

- [An Effectiveness Study Across Baseline and Learning-based Force Estimation Methods on the da Vinci Research Kit Si System](https://mlsanity.com/paper/2405.07453/) (2024)

## Abstract (arXiv)

Force estimation using neural networks is a promising approach to enable haptic feedback in minimally invasive surgical robots without end-effector force sensors. Various network architectures have been proposed, but none have been tested in real time with surgical-like manipulations. Thus, questions remain about the real-time transparency and stability of force feedback from neural network-based force estimates. We characterize the real-time impedance transparency and stability of force feedback rendered on a da Vinci Research Kit teleoperated surgical robot using neural networks with vision-only, state-only, and state and vision inputs. Networks were trained on an existing dataset of teleoperated manipulations without force feedback. To measure real-time stability and transparency during teleoperation with force feedback to the operator, we modeled a one-degree-of-freedom human and surgeon-side manipulandum that moved the patient-side robot to perform manipulations on silicone artificial tissue over various robot and camera configurations, and tools. We found that the networks using state inputs displayed more transparent impedance than a vision-only network. However, state-based networks displayed large instability when used to provide force feedback during lateral manipulation of the silicone. In contrast, the vision-only network showed consistent stability in all the evaluated directions. We confirmed the performance of the vision-only network for real-time force feedback in a demonstration with a human teleoperator.