# CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions

arXiv: https://arxiv.org/abs/1910.09031 · HTML: https://mlsanity.com/paper/1910.09031/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2019.

## Built on by (most cited first, 30 of 40)

- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark](https://mlsanity.com/paper/2109.14956/) (2021)
- [LoViT: Long Video Transformer for Surgical Phase Recognition](https://mlsanity.com/paper/2305.08989/) (2023)
- [CholecTriplet2021: A benchmark challenge for surgical action triplet recognition](https://mlsanity.com/paper/2204.04746/) (2022)
- [Multi-Task Temporal Convolutional Networks for Joint Recognition of Surgical Phases and Steps in Gastric Bypass Procedures](https://mlsanity.com/paper/2102.12218/) (2021)
- [EgoLM: Multi-Modal Language Model of Egocentric Motions](https://mlsanity.com/paper/2409.18127/) (2024)
- [Rendezvous in Time: An Attention-based Temporal Fusion approach for Surgical Triplet Recognition](https://mlsanity.com/paper/2211.16963/) (2022)
- [CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detection](https://mlsanity.com/paper/2302.06294/) (2023)
- [Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions](https://mlsanity.com/paper/2307.09548/) (2023)
- [The Impact of Machine Learning on 2D/3D Registration for Image-guided Interventions: A Systematic Review and Perspective](https://mlsanity.com/paper/2108.02238/) (2021)
- [CholecInstanceSeg: A Tool Instance Segmentation Dataset for Laparoscopic Surgery](https://mlsanity.com/paper/2406.16039/) (2024)
- [Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoning](https://mlsanity.com/paper/2312.06829/) (2023)
- [VidLPRO: A underlineVideo-underlineLanguage underlinePre-training Framework for underlineRobotic and Laparoscopic Surgery](https://mlsanity.com/paper/2409.04732/) (2024)
- [Simulation-to-Real domain adaptation with teacher-student learning for endoscopic instrument segmentation](https://mlsanity.com/paper/2103.01593/) (2021)
- [SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Models](https://mlsanity.com/paper/2412.02332/) (2024)
- [Unsupervised domain adaptation for clinician pose estimation and instance segmentation in the operating room](https://mlsanity.com/paper/2108.11801/) (2021)
- [Temporally Constrained Neural Networks (TCNN): A framework for semi-supervised video semantic segmentation](https://mlsanity.com/paper/2112.13815/) (2021)
- [ST(OR)2: Spatio-Temporal Object Level Reasoning for Activity Recognition in the Operating Room](https://mlsanity.com/paper/2312.12250/) (2023)
- [SR-Mamba: Effective Surgical Phase Recognition with State Space Model](https://mlsanity.com/paper/2407.08333/) (2024)
- [Surgical Text-to-Image Generation](https://mlsanity.com/paper/2407.09230/) (2024)
- [Adaptation of Surgical Activity Recognition Models Across Operating Rooms](https://mlsanity.com/paper/2207.03083/) (2022)
- [Current State of Community-Driven Radiological AI Deployment in Medical Imaging](https://mlsanity.com/paper/2212.14177/) (2022)
- [Weakly Supervised Temporal Convolutional Networks for Fine-grained Surgical Activity Recognition](https://mlsanity.com/paper/2302.10834/) (2023)
- [Multitask Learning in Minimally Invasive Surgical Vision: A Review](https://mlsanity.com/paper/2401.08256/) (2024)
- [Why Deep Surgical Models Fail?: Revisiting Surgical Action Triplet Recognition through the Lens of Robustness](https://mlsanity.com/paper/2209.08647/) (2022)
- [ACT-Net: Anchor-context Action Detection in Surgery Videos](https://mlsanity.com/paper/2310.03377/) (2023)
- [SPRMamba: Surgical Phase Recognition for Endoscopic Submucosal Dissection with Mamba](https://mlsanity.com/paper/2409.12108/) (2024)
- [TEMSET-24K: Densely Annotated Dataset for Indexing Multipart Endoscopic Videos using Surgical Timeline Segmentation](https://mlsanity.com/paper/2502.06708/) (2025)
- [Hierarchical Context Transformer for Multi-level Semantic Scene Understanding](https://mlsanity.com/paper/2502.15184/) (2025)
- [fine-CLIP: Enhancing Zero-Shot Fine-Grained Surgical Action Recognition with Vision-Language Models](https://mlsanity.com/paper/2503.19670/) (2025)

## Abstract (arXiv)

Data-driven computational approaches have evolved to enable extraction of information from medical images with a reliability, accuracy and speed which is already transforming their interpretation and exploitation in clinical practice. While similar benefits are longed for in the field of interventional imaging, this ambition is challenged by a much higher heterogeneity. Clinical workflows within interventional suites and operating theatres are extremely complex and typically rely on poorly integrated intra-operative devices, sensors, and support infrastructures. Taking stock of some of the most exciting developments in machine learning and artificial intelligence for computer assisted interventions, we highlight the crucial need to take context and human factors into account in order to address these challenges. Contextual artificial intelligence for computer assisted intervention, or CAI4CAI, arises as an emerging opportunity feeding into the broader field of surgical data science. Central challenges being addressed in CAI4CAI include how to integrate the ensemble of prior knowledge and instantaneous sensory information from experts, sensors and actuators; how to create and communicate a faithful and actionable shared representation of the surgery among a mixed human-AI actor team; how to design interventional systems and associated cognitive shared control schemes for online uncertainty-aware collaborative decision making ultimately producing more precise and reliable interventions.