# Adaptive Graph Learning from Spatial Information for Surgical Workflow Anticipation

arXiv: https://arxiv.org/abs/2412.06454 · HTML: https://mlsanity.com/paper/2412.06454/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 59 references that are in the map.
- First posted on arXiv 9 Dec 2024.

## Builds on (most cited first, 26 of 26)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [Learning Latent Permutations with Gumbel-Sinkhorn Networks](https://mlsanity.com/paper/1802.08665/) (2018)
- [Surgical Data Science: Enabling Next-Generation Surgery](https://mlsanity.com/paper/1701.06482/) (2017)
- [On the Design Fundamentals of Diffusion Models: A Survey](https://mlsanity.com/paper/2306.04542/) (2023)
- [Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks](https://mlsanity.com/paper/1802.08774/) (2018)
- [Learning to Anticipate Egocentric Actions by Imagination](https://mlsanity.com/paper/2101.04924/) (2021)
- [Learning and Reasoning with the Graph Structure Representation in Robotic Surgery](https://mlsanity.com/paper/2007.03357/) (2020)
- [RSDNet: Learning to Predict Remaining Surgery Duration from Laparoscopic Videos Without Manual Annotations](https://mlsanity.com/paper/1802.03243/) (2018)
- [A deep learning framework for quality assessment and restoration in video endoscopy](https://mlsanity.com/paper/1904.07073/) (2019)
- [Context Understanding in Computer Vision: A Survey](https://mlsanity.com/paper/2302.05011/) (2023)
- [Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery](https://mlsanity.com/paper/2011.01619/) (2020)
- [Geometric Features Informed Multi-person Human-object Interaction Recognition in Videos](https://mlsanity.com/paper/2207.09425/) (2022)
- [CataNet: Predicting remaining cataract surgery duration](https://mlsanity.com/paper/2106.11048/) (2021)
- [Rethinking Anticipation Tasks: Uncertainty-aware Anticipation of Sparse Surgical Instrument Usage for Context-aware Assistance](https://mlsanity.com/paper/2007.00548/) (2020)
- [Recognition and Prediction of Surgical Gestures and Trajectories Using Transformer Models in Robot-Assisted Surgery](https://mlsanity.com/paper/2212.01683/) (2022)
- [Towards Generalizable Surgical Activity Recognition Using Spatial Temporal Graph Convolutional Networks](https://mlsanity.com/paper/2001.03728/) (2020)
- [Elsa: Energy-based learning for semi-supervised anomaly detection](https://mlsanity.com/paper/2103.15296/) (2021)
- [Towards Graph Representation Learning Based Surgical Workflow Anticipation](https://mlsanity.com/paper/2208.03824/) (2022)
- [Predicting the Timing of Camera Movements From the Kinematics of Instruments in Robotic-Assisted Surgery Using Artificial Neural Networks](https://mlsanity.com/paper/2109.11192/) (2021)

## Built on by (most cited first, 2 of 2)

- [CoStoDet-DDPM: Collaborative Training of Stochastic and Deterministic Models Improves Surgical Workflow Anticipation and Recognition](https://mlsanity.com/paper/2503.10216/) (2025)
- [Cross-view Multimodal Vision-Based Assessment Framework for Traditional Chinese Medicine Rehabilitation Training](https://mlsanity.com/paper/2606.28104/) (2026)

## Abstract (arXiv)

Surgical workflow anticipation is the task of predicting the timing of relevant surgical events from live video data, which is critical in Robotic-Assisted Surgery (RAS). Accurate predictions require the use of spatial information to model surgical interactions. However, current methods focus solely on surgical instruments, assume static interactions between instruments, and only anticipate surgical events within a fixed time horizon. To address these challenges, we propose an adaptive graph learning framework for surgical workflow anticipation based on a novel spatial representation, featuring three key innovations. First, we introduce a new representation of spatial information based on bounding boxes of surgical instruments and targets, including their detection confidence levels. These are trained on additional annotations we provide for two benchmark datasets. Second, we design an adaptive graph learning method to capture dynamic interactions. Third, we develop a multi-horizon objective that balances learning objectives for different time horizons, allowing for unconstrained predictions. Evaluations on two benchmarks reveal superior performance in short-to-mid-term anticipation, with an error reduction of approximately 3% for surgical phase anticipation and 9% for remaining surgical duration anticipation. These performance improvements demonstrate the effectiveness of our method and highlight its potential for enhancing preparation and coordination within the RAS team. This can improve surgical safety and the efficiency of operating room usage.