mlsanityData to 14 Oct 2026

Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos

arXiv 2109.03223 ↗First posted 7 Sep 2021cs.CVcited by 141

Not on the map. Why?

Abstract (arXiv)

Out of all existing frameworks for surgical workflow analysis in endoscopic videos, action triplet recognition stands out as the only one aiming to provide truly fine-grained and comprehensive information on surgical activities. This information, presented as <instrument, verb, target> combinations, is highly challenging to be accurately identified. Triplet components can be difficult to recognize individually; in this task, it requires not only performing recognition simultaneously for all three triplet components, but also correctly establishing the data association between them. To achieve this task, we introduce our new model, the Rendezvous (RDV), which recognizes triplets directly from surgical videos by leveraging attention at two different levels. We first introduce a new form of spatial attention to capture individual action triplet components in a scene; called Class Activation Guided Attention Mechanism (CAGAM). This technique focuses on the recognition of verbs and targets using activations resulting from instruments. To solve the association problem, our RDV model adds a new form of semantic attention inspired by Transformer networks; called Multi-Head of Mixed Attention (MHMA). This technique uses several cross and self attentions to effectively capture relationships between instruments, verbs, and targets. We also introduce CholecT50 - a dataset of 50 endoscopic videos in which every frame has been annotated with labels from 100 triplet classes. Our proposed RDV model significantly improves the triplet prediction mean AP by over 9% compared to the state-of-the-art methods on this dataset.

Key facts. Cited by 141 papers in the map (data through 14 Oct 2026). Builds on 48 of its 88 references that are in the map. First posted on arXiv 7 Sep 2021.

Builds on

48 of 88 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  6. 2017Non-local Neural Networkscited by 2,293
  7. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  8. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,558
  9. 2018Attention U-Net: Learning Where to Look for the Pancreascited by 1,435
  10. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  11. 2018Dual Attention Network for Scene Segmentationcited by 1,003
  12. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 718
36 more
  1. 2016EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videoscited by 367
  2. 2021Medical Transformer: Gated Axial-Attention for Medical Image Segmentationcited by 320
  3. 2018Video Action Transformer Networkcited by 316
  4. 2017Learning to Detect Human-Object Interactionscited by 240
  5. 2016Temporal Convolutional Networks: A Unified Approach to Action Segmentationcited by 199
  6. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  7. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  8. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 135
  9. 2017Surgical Data Science: Enabling Next-Generation Surgerycited by 126
  10. 20202018 Robotic Scene Segmentation Challengecited by 124
  11. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  12. 2021Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformercited by 94
  13. 2020Actor-Transformers for Group Activity Recognitioncited by 84
  14. 2020Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Tripletscited by 66
  15. 2021OperA: Attention-Regularized Transformers for Surgical Phase Recognitioncited by 60
  16. 2021CPTR: Full Transformer Network for Image Captioningcited by 55
  17. 2019Deep Contextual Attention for Human-Object Interaction Detectioncited by 54
  18. 2018DeepPhase: Surgical Phase Recognition in CATARACTS Videoscited by 47
  19. 2021Multi-Task Temporal Convolutional Networks for Joint Recognition of Surgical Phases and Steps in Gastric Bypass Procedurescited by 42
  20. 2019CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventionscited by 40
  21. 2021The SARAS Endoscopic Surgeon Action Detection (ESAD) dataset: Challenges and methodscited by 38
  22. 2018Weakly Supervised Convolutional LSTM Approach for Tool Tracking in Laparoscopic Videoscited by 35
  23. 2017ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Toolscited by 30
  24. 2019Saliency Guided Self-attention Network for Weakly and Semi-supervised Semantic Segmentationcited by 28
  25. 2016Recognizing Surgical Activities with Recurrent Neural Networkscited by 21
  26. 2018Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognitioncited by 21
  27. 2019Saliency-Guided Attention Network for Image-Sentence Matchingcited by 21
  28. 2018Weakly-Supervised Learning for Tool Localization in Laparoscopic Videoscited by 20
  29. 2018Surgical Activity Recognition in Robot-Assisted Radical Prostatectomy using Deep Learningcited by 18
  30. 2018Temporal coherence-based self-supervised learning for laparoscopic workflow analysiscited by 16
  31. 2018Detecting Visual Relationships Using Box Attentioncited by 16
  32. 2021Learning Domain Adaptation with Model Calibration for Surgical Report Generation in Robotic Surgerycited by 16
  33. 2016Learning Models for Actions and Person-Object Interactions with Transfer to Question Answeringcited by 15
  34. 2019Offline identification of surgical deviations in laparoscopic rectopexycited by 12
  35. 2021End-to-End Attention-based Image Captioningcited by 4
  36. 2016Temporal Attention Model for Neural Machine Translationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 14; 2023: 18; 2024: 30; 2025: 42; 2026: 37.

141 papers.

Sort:

By area:

  1. 2023Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-traincited by 75
  2. 2023Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lecturescited by 68
  3. 2024Endora: Video Generation Models as Endoscopy Simulatorscited by 55
  4. 2024Pixel-Wise Recognition for Holistic Surgical Scene Understandingcited by 49
  5. 2022Dissecting Self-Supervised Learning Methods for Surgical Computer Visioncited by 48
  6. 2022CholecTriplet2021: A benchmark challenge for surgical action triplet recognitioncited by 44
  7. 2024HecVL: Hierarchical Video-Language Pretraining for Zero-shot Surgical Phase Recognitioncited by 42
  8. 2023Advancing Surgical VQA with Scene Graph Knowledgecited by 41
  9. 2023Challenges in Multi-centric Generalization: Phase and Step Recognition in Roux-en-Y Gastric Bypass Surgerycited by 40
  10. 2024Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentationcited by 39
  11. 2022Towards Holistic Surgical Scene Understandingcited by 33
  12. 2024OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understandingcited by 32
38 more
  1. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligencecited by 31
  2. 2025MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environmentscited by 28
  3. 2022Exploring Intra- and Inter-Video Relation for Surgical Semantic Scene Segmentationcited by 27
  4. 2022Rendezvous in Time: An Attention-based Temporal Fusion approach for Surgical Triplet Recognitioncited by 27
  5. 2022Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasetscited by 23
  6. 2023CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detectioncited by 23
  7. 2023Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactionscited by 23
  8. 2023LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Roomscited by 21
  9. 2025LEMON: A Large Endoscopic MONocular Dataset and Foundation Model for Perception in Surgical Settingscited by 20
  10. 2023CholecTrack20: A Multi-Perspective Tracking Dataset for Surgical Toolscited by 17
  11. 2024Large Language Model Benchmarks in Medical Taskscited by 17
  12. 2024CholecInstanceSeg: A Tool Instance Segmentation Dataset for Laparoscopic Surgerycited by 16
  13. 2025SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learningcited by 16
  14. 2024General-purpose foundation models for increased autonomy in robot-assisted surgerycited by 15
  15. 2024Bora: Biomedical Generalist Video Generation Modelcited by 15
  16. 2024SurgSora: Object-Aware Diffusion Model for Controllable Surgical Video Generationcited by 15
  17. 2024ORacle: Large Vision-Language Models for Knowledge-Guided Holistic OR Domain Modelingcited by 14
  18. 2022Concept Graph Neural Networks for Surgical Video Understandingcited by 13
  19. 2024VidLPRO: A underlineVideo-underlineLanguage underlinePre-training Framework for underlineRobotic and Laparoscopic Surgerycited by 12
  20. 2024EndoOmni: Zero-Shot Cross-Dataset Depth Estimation in Endoscopy by Robust Self-Learning from Noisy Labelscited by 12
  21. 2023Syn-Mediverse: A Multimodal Synthetic Dataset for Intelligent Scene Understanding of Healthcare Facilitiescited by 11
  22. 2024VISAGE: Video Synthesis using Action Graphs for Surgerycited by 11
  23. 2025EndoMamba: An Efficient Foundation Model for Endoscopic Videos via Hierarchical Pre-trainingcited by 11
  24. 2025EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understandingcited by 11
  25. 2025Large-scale Self-supervised Video Foundation Model for Intelligent Surgerycited by 10
  26. 2023Self-distillation for surgical action recognitioncited by 9
  27. 2025SurgBench: A Unified Large-Scale Benchmark for Surgical Video Analysiscited by 9
  28. 2025MedGen: Unlocking Medical Video Generation by Scaling Granularly-annotated Medical Videoscited by 9
  29. 2023Whether and When does Endoscopy Domain Pretraining Make Sense?cited by 8
  30. 2024Instrument-tissue Interaction Detection Framework for Surgical Video Understandingcited by 8
  31. 2024Interactive Generation of Laparoscopic Videos with Diffusion Modelscited by 8
  32. 2024Data Augmentation for Surgical Scene Segmentation with Anatomy-Aware Diffusion Modelscited by 8
  33. 2024Optical Flow Representation Alignment Mamba Diffusion Model for Medical Video Generationcited by 8
  34. 2024SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Modelscited by 8
  35. 2025HieraSurg: Hierarchy-Aware Diffusion Model for Surgical Video Generationcited by 8
  36. 2025MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understandingcited by 8
  37. 2022Video-based Surgical Skills Assessment using Long term Tool Trackingcited by 7
  38. 2023Navigating the Synthetic Realm: Harnessing Diffusion-based Models for Laparoscopic Text-to-Image Generationcited by 7

50 most cited of 141 All 141 (JSON)