# Real-Time Multimodal Activity-Aware Error Detection in Robot-Assisted Surgery

arXiv: https://arxiv.org/abs/2606.23593 · HTML: https://mlsanity.com/paper/2606.23593/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Temporal Convolutional Networks: A Unified Approach to Action Segmentation](https://mlsanity.com/paper/1608.08242/) (2016)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [Video-based surgical skill assessment using 3D convolutional neural networks](https://mlsanity.com/paper/1903.02306/) (2019)
- [SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge](https://mlsanity.com/paper/2401.00496/) (2023)
- [CholecTriplet2021: A benchmark challenge for surgical action triplet recognition](https://mlsanity.com/paper/2204.04746/) (2022)
- [Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery](https://mlsanity.com/paper/2405.10948/) (2024)
- [Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation](https://mlsanity.com/paper/2410.00263/) (2024)
- [daVinciNet: Joint Prediction of Motion and Surgical State in Robot-Assisted Surgery](https://mlsanity.com/paper/2009.11937/) (2020)
- [Real-Time Context-aware Detection of Unsafe Events in Robot-Assisted Surgery](https://mlsanity.com/paper/2005.03611/) (2020)
- [Multimodal Transformers for Real-Time Surgical Activity Prediction](https://mlsanity.com/paper/2403.06705/) (2024)
- [Adverse Events in Robotic Surgery: A Retrospective Study of 14 Years of FDA Data](https://mlsanity.com/paper/1507.03518/) (2015)
- [Analysis of Executional and Procedural Errors in Dry-lab Robotic Surgery Experiments](https://mlsanity.com/paper/2106.11962/) (2021)
- [COMPASS: A Formal Framework and Aggregate Dataset for Generalized Surgical Procedure Modeling](https://mlsanity.com/paper/2209.06424/) (2022)
- [Context-aware Monitoring in Robotic Surgery](https://mlsanity.com/paper/1901.09802/) (2019)
- [Deep Multimodal Fusion for Surgical Feedback Classification](https://mlsanity.com/paper/2312.03231/) (2023)
- [Think Step by Step: Chain-of-Gesture Prompting for Error Detection in Robotic Surgical Videos](https://mlsanity.com/paper/2406.19217/) (2024)
- [Towards Surgical Context Inference and Translation to Gestures](https://mlsanity.com/paper/2302.14237/) (2023)
- [SEDMamba: Enhancing Selective State Space Modelling with Bottleneck Mechanism and Fine-to-Coarse Temporal Fusion for Efficient Error Detection in Robot-Assisted Surgery](https://mlsanity.com/paper/2406.15920/) (2024)
- [CARES: Collaborative Agentic Reasoning for Error Detection in Surgery](https://mlsanity.com/paper/2508.08764/) (2025)
- [Robotic Scene Segmentation with Memory Network for Runtime Surgical Context Inference](https://mlsanity.com/paper/2308.12789/) (2023)
- [Towards Interpretable Motion-level Skill Assessment in Robotic Surgery](https://mlsanity.com/paper/2311.05838/) (2023)
- [Evaluating the Task Generalization of Temporal Convolutional Networks for Surgical Gesture and Motion Recognition using Kinematic Data](https://mlsanity.com/paper/2306.16577/) (2023)
- [MiDAS: A Multimodal Data Acquisition System and Dataset for Robot-Assisted Minimally Invasive Surgery](https://mlsanity.com/paper/2602.12407/) (2026)

## Abstract (arXiv)

Robot-assisted minimally invasive surgery improves surgical precision but introduces complexity, making technical error detection essential for ensuring patient safety. Current executional error detection methods using video data often overlook fine-grained contextual descriptions of activities and error types within the hierarchical structure of surgical procedures. They also under-utilize complementary multimodal information. We propose a unified framework for executional error detection that leverages multimodal input, including video, kinematics, and descriptive textual prompts. Through activity prompting, we integrate descriptive language in gesture-level activities, instrument-object interactions, and error definitions. We also introduce activity-aware visual embeddings derived from vision encoders pretrained on surgical activity labels to compare the effectiveness of contrastive language-image embeddings with traditional image-based embeddings for error detection. By seamlessly integrating kinematic data with video and textual modalities, our framework significantly improves error detection performance. Achieving up to 5% and 16.6% F1 score improvements over state-of-the-art baselines on the JIGSAWS and SAR-RARP50 datasets, respectively, we demonstrate the value of combining curated textual prompts with multimodal data for accurate error detection.