# Robotic Scene Segmentation with Memory Network for Runtime Surgical Context Inference

arXiv: https://arxiv.org/abs/2308.12789 · HTML: https://mlsanity.com/paper/2308.12789/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 24 Aug 2023.

## Builds on (most cited first, 25 of 25)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation](https://mlsanity.com/paper/2106.05210/) (2021)
- [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019)
- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [2018 Robotic Scene Segmentation Challenge](https://mlsanity.com/paper/2001.11190/) (2020)
- [Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning](https://mlsanity.com/paper/1803.01207/) (2018)
- [ISINet: An Instance-Based Approach for Surgical Instrument Segmentation](https://mlsanity.com/paper/2007.05533/) (2020)
- [Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets](https://mlsanity.com/paper/2007.05405/) (2020)
- [Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video](https://mlsanity.com/paper/1907.07899/) (2019)
- [CholecTriplet2021: A benchmark challenge for surgical action triplet recognition](https://mlsanity.com/paper/2204.04746/) (2022)
- [TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/2202.08453/) (2022)
- [Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery](https://mlsanity.com/paper/1805.02475/) (2018)
- [Real-Time Segmentation of Non-Rigid Surgical Tools based on Deep Learning and Tracking](https://mlsanity.com/paper/2009.03016/) (2020)
- [Real-Time Context-aware Detection of Unsafe Events in Robot-Assisted Surgery](https://mlsanity.com/paper/2005.03611/) (2020)
- [Cooperative Assistance in Robotic Surgery through Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2110.04857/) (2021)
- [Analysis of Executional and Procedural Errors in Dry-lab Robotic Surgery Experiments](https://mlsanity.com/paper/2106.11962/) (2021)
- [COMPASS: A Formal Framework and Aggregate Dataset for Generalized Surgical Procedure Modeling](https://mlsanity.com/paper/2209.06424/) (2022)
- [Context-aware Monitoring in Robotic Surgery](https://mlsanity.com/paper/1901.09802/) (2019)
- [Runtime Detection of Executional Errors in Robot-Assisted Surgery](https://mlsanity.com/paper/2203.00737/) (2022)
- [A Learning-Driven Framework with Spatial Optimization For Surgical Suture Thread Reconstruction and Autonomous Grasping Under Multiple Topologies and Environmental Noises](https://mlsanity.com/paper/2007.00920/) (2020)
- [Towards Surgical Context Inference and Translation to Gestures](https://mlsanity.com/paper/2302.14237/) (2023)

## Built on by (most cited first, 2 of 2)

- [Multi-Modal Gesture Recognition from Video and Surgical Tool Pose Information via Motion Invariants](https://mlsanity.com/paper/2503.15647/) (2025)
- [Real-Time Multimodal Activity-Aware Error Detection in Robot-Assisted Surgery](https://mlsanity.com/paper/2606.23593/) (2026)

## Abstract (arXiv)

Surgical context inference has recently garnered significant attention in robot-assisted surgery as it can facilitate workflow analysis, skill assessment, and error detection. However, runtime context inference is challenging since it requires timely and accurate detection of the interactions among the tools and objects in the surgical scene based on the segmentation of video data. On the other hand, existing state-of-the-art video segmentation methods are often biased against infrequent classes and fail to provide temporal consistency for segmented masks. This can negatively impact the context inference and accurate detection of critical states. In this study, we propose a solution to these challenges using a Space Time Correspondence Network (STCN). STCN is a memory network that performs binary segmentation and minimizes the effects of class imbalance. The use of a memory bank in STCN allows for the utilization of past image and segmentation information, thereby ensuring consistency of the masks. Our experiments using the publicly available JIGSAWS dataset demonstrate that STCN achieves superior segmentation performance for objects that are difficult to segment, such as needle and thread, and improves context inference compared to the state-of-the-art. We also demonstrate that segmentation and context inference can be performed at runtime without compromising performance.