# Instrument-tissue Interaction Detection Framework for Surgical Video Understanding arXiv: https://arxiv.org/abs/2404.00322 ยท HTML: https://mlsanity.com/paper/2404.00322/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 38 references that are in the map. - First posted on arXiv 30 Mar 2024. ## Builds on (most cited first, 28 of 28) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017) - [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016) - [Temporal Convolutional Networks: A Unified Approach to Action Segmentation](https://mlsanity.com/paper/1608.08242/) (2016) - [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015) - [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018) - [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021) - [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021) - [End-to-End Human Object Interaction Detection with HOI Transformer](https://mlsanity.com/paper/2103.04503/) (2021) - [iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection](https://mlsanity.com/paper/1808.10437/) (2018) - [Surgical Data Science: Enabling Next-Generation Surgery](https://mlsanity.com/paper/1701.06482/) (2017) - [CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80](https://mlsanity.com/paper/2012.12453/) (2020) - [Spatially Conditioned Graphs for Detecting Human-Object Interactions](https://mlsanity.com/paper/2012.06060/) (2020) - [Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformer](https://mlsanity.com/paper/2112.01838/) (2021) - [Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets](https://mlsanity.com/paper/2007.05405/) (2020) - [Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Room](https://mlsanity.com/paper/2005.03501/) (2020) - [Multi-Task Temporal Convolutional Networks for Joint Recognition of Surgical Phases and Steps in Gastric Bypass Procedures](https://mlsanity.com/paper/2102.12218/) (2021) - [The SARAS Endoscopic Surgeon Action Detection (ESAD) dataset: Challenges and methods](https://mlsanity.com/paper/2104.03178/) (2021) - [Towards Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2212.04582/) (2022) - [Learning and Reasoning with the Graph Structure Representation in Robotic Surgery](https://mlsanity.com/paper/2007.03357/) (2020) - [Global-Reasoned Multi-Task Learning Model for Surgical Scene Understanding](https://mlsanity.com/paper/2201.11957/) (2022) - [MIcro-Surgical Anastomose Workflow recognition challenge report](https://mlsanity.com/paper/2103.13111/) (2021) - [Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection](https://mlsanity.com/paper/2008.00936/) (2020) - [Recognizing Surgical Activities with Recurrent Neural Networks](https://mlsanity.com/paper/1606.06329/) (2016) - [Learning Domain Adaptation with Model Calibration for Surgical Report Generation in Robotic Surgery](https://mlsanity.com/paper/2103.17120/) (2021) - [ESAD: Endoscopic Surgeon Action Detection Dataset](https://mlsanity.com/paper/2006.07164/) (2020) - [MURPHY: Relations Matter in Surgical Workflow Analysis](https://mlsanity.com/paper/2212.12719/) (2022) ## Built on by (most cited first, 8 of 8) - [Memory-Augmented Multimodal LLMs for Surgical VQA via Self-Contained Inquiry](https://mlsanity.com/paper/2411.10937/) (2024) - [ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detection](https://mlsanity.com/paper/2506.01130/) (2025) - [Decoding the Surgical Scene: A Scoping Review of Scene Graphs in Surgery](https://mlsanity.com/paper/2509.20941/) (2025) - [Hierarchical Context Transformer for Multi-level Semantic Scene Understanding](https://mlsanity.com/paper/2502.15184/) (2025) - [Multi-scale Temporal Prediction via Incremental Generation and Multi-agent Collaboration](https://mlsanity.com/paper/2509.17429/) (2025) - [Tracking spatial temporal details in ultrasound long video via wavelet analysis and memory bank](https://mlsanity.com/paper/2512.15066/) (2025) - [Latent-Action-Guided Video-Language Feature Learning for Surgical Instrument-Tissue Interaction Recognition](https://mlsanity.com/paper/2607.19889/) (2026) - [Beyond Instrument Motion: Recognizing Tissue Tension Toward Surgical Skill Assessment](https://mlsanity.com/paper/2608.17935/) (2026) ## Abstract (arXiv) Instrument-tissue interaction detection task, which helps understand surgical activities, is vital for constructing computer-assisted surgery systems but with many challenges. Firstly, most models represent instrument-tissue interaction in a coarse-grained way which only focuses on classification and lacks the ability to automatically detect instruments and tissues. Secondly, existing works do not fully consider relations between intra- and inter-frame of instruments and tissues. In the paper, we propose to represent instrument-tissue interaction as quintuple and present an Instrument-Tissue Interaction Detection Network (ITIDNet) to detect the quintuple for surgery videos understanding. Specifically, we propose a Snippet Consecutive Feature (SCF) Layer to enhance features by modeling relationships of proposals in the current frame using global context information in the video snippet. We also propose a Spatial Corresponding Attention (SCA) Layer to incorporate features of proposals between adjacent frames through spatial encoding. To reason relationships between instruments and tissues, a Temporal Graph (TG) Layer is proposed with intra-frame connections to exploit relationships between instruments and tissues in the same frame and inter-frame connections to model the temporal information for the same instance. For evaluation, we build a cataract surgery video (PhacoQ) dataset and a cholecystectomy surgery video (CholecQ) dataset. Experimental results demonstrate the promising performance of our model, which outperforms other state-of-the-art models on both datasets.