mlsanityData to 7 Oct 2026

Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets

arXiv 2007.05405 ↗First posted 10 Jul 2020eess.IV cs.CV cs.LGcited by 66

Not on the map. Why?

Abstract (arXiv)

Recognition of surgical activity is an essential component to develop context-aware decision support for the operating room. In this work, we tackle the recognition of fine-grained activities, modeled as action triplets <instrument, verb, target> representing the tool activity. To this end, we introduce a new laparoscopic dataset, CholecT40, consisting of 40 videos from the public dataset Cholec80 in which all frames have been annotated using 128 triplet classes. Furthermore, we present an approach to recognize these triplets directly from the video data. It relies on a module called Class Activation Guide (CAG), which uses the instrument activation maps to guide the verb and target recognition. To model the recognition of multiple triplets in the same frame, we also propose a trainable 3D Interaction Space, which captures the associations between the triplet components. Finally, we demonstrate the significance of these contributions via several ablation studies and comparisons to baselines on CholecT40.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 10; 2023: 10; 2024: 11; 2025: 19; 2026: 13.

66 papers.

Sort:

By area:

  1. 2021Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videoscited by 139
  2. 2024Pixel-Wise Recognition for Holistic Surgical Scene Understandingcited by 48
  3. 2022CholecTriplet2021: A benchmark challenge for surgical action triplet recognitioncited by 44
  4. 2023Challenges in Multi-centric Generalization: Phase and Step Recognition in Roux-en-Y Gastric Bypass Surgerycited by 40
  5. 2022Towards Holistic Surgical Scene Understandingcited by 33
  6. 2024OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understandingcited by 31
  7. 2025MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environmentscited by 28
  8. 2022Rendezvous in Time: An Attention-based Temporal Fusion approach for Surgical Triplet Recognitioncited by 27
  9. 2025Scaling up self-supervised learning for improved surgical foundation modelscited by 25
  10. 2022Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasetscited by 23
  11. 2023Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactionscited by 23
  12. 2023CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detectioncited by 22
38 more
  1. 2023LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Roomscited by 21
  2. 2022FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentationcited by 20
  3. 2023CholecTrack20: A Multi-Perspective Tracking Dataset for Surgical Toolscited by 17
  4. 2022Concept Graph Neural Networks for Surgical Video Understandingcited by 13
  5. 2025EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understandingcited by 11
  6. 2025Large-scale Self-supervised Video Foundation Model for Intelligent Surgerycited by 10
  7. 2023Self-distillation for surgical action recognitioncited by 9
  8. 2024VISAGE: Video Synthesis using Action Graphs for Surgerycited by 9
  9. 2023Preserving Privacy in Surgical Video Analysis Using Artificial Intelligence: A Deep Learning Classifier to Identify Out-of-Body Scenes in Endoscopic Videoscited by 8
  10. 2023Whether and When does Endoscopy Domain Pretraining Make Sense?cited by 8
  11. 2024Instrument-tissue Interaction Detection Framework for Surgical Video Understandingcited by 8
  12. 2025Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Reviewcited by 8
  13. 2021SUPR-GAN: SUrgical PRediction GAN for Event Anticipation in Laparoscopic and Robotic Surgerycited by 7
  14. 2024Optical Flow Representation Alignment Mamba Diffusion Model for Medical Video Generationcited by 7
  15. 2021Temporally Constrained Neural Networks (TCNN): A framework for semi-supervised video semantic segmentationcited by 6
  16. 2024Surgical Text-to-Image Generationcited by 6
  17. 2023Weakly Supervised Temporal Convolutional Networks for Fine-grained Surgical Activity Recognitioncited by 5
  18. 2024Multitask Learning in Minimally Invasive Surgical Vision: A Reviewcited by 5
  19. 2024Hypergraph-Transformer (HGT) for Interactive Event Prediction in Laparoscopic and Robotic Surgerycited by 5
  20. 2022Why Deep Surgical Models Fail?: Revisiting Surgical Action Triplet Recognition through the Lens of Robustnesscited by 4
  21. 2022Task-Aware Asynchronous Multi-Task Model with Class Incremental Contrastive Learning for Surgical Scene Understandingcited by 4
  22. 2024Surgical Triplet Recognition via Diffusion Modelcited by 4
  23. 2025ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detectioncited by 4
  24. 2025MEJO: MLLM-Engaged Surgical Triplet Recognition via Inter- and Intra-Task Joint Optimizationcited by 4
  25. 2022MURPHY: Relations Matter in Surgical Workflow Analysiscited by 3
  26. 2025Decoding the Surgical Scene: A Scoping Review of Scene Graphs in Surgerycited by 3
  27. 2022Live Laparoscopic Video Retrieval with Compressed Uncertaintycited by 2
  28. 2023Robotic Scene Segmentation with Memory Network for Runtime Surgical Context Inferencecited by 2
  29. 2024Strategies to Improve Real-World Applicability of Laparoscopic Anatomy Segmentation Modelscited by 2
  30. 2024Artificial Intelligence for Biomedical Video Generationcited by 2
  31. 2025Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Datacited by 2
  32. 2025TEMSET-24K: Densely Annotated Dataset for Indexing Multipart Endoscopic Videos using Surgical Timeline Segmentationcited by 2
  33. 2025Hierarchical Context Transformer for Multi-level Semantic Scene Understandingcited by 2
  34. 2025DARIL: When Imitation Learning outperforms Reinforcement Learning in Surgical Action Planningcited by 1
  35. 2025Surgical Video Understanding with Label Interpolationcited by 1
  36. 2025Grounding Surgical Action Triplets with Instrument Instance Segmentation: A Dataset and Target-Aware Fusion Approachcited by 1
  37. 2026SurgSigma: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligencecited by 1
  38. 2026Scaling Video Pretraining for Surgical Foundation Modelscited by 1

50 most cited of 66 All 66 (JSON)