# CholecTriplet2021: A benchmark challenge for surgical action triplet recognition

arXiv: https://arxiv.org/abs/2204.04746 · HTML: https://mlsanity.com/paper/2204.04746/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 71 references that are in the map.
- First posted on arXiv 10 Apr 2022.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [CSPNet: A New Backbone that can Enhance Learning Capability of CNN](https://mlsanity.com/paper/1911.11929/) (2019)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [Coordinate Attention for Efficient Mobile Network Design](https://mlsanity.com/paper/2103.02907/) (2021)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks](https://mlsanity.com/paper/2003.10751/) (2020)
- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [Surgical Data Science: Enabling Next-Generation Surgery](https://mlsanity.com/paper/1701.06482/) (2017)
- [Visual Translation Embedding Network for Visual Relation Detection](https://mlsanity.com/paper/1702.08319/) (2017)
- [Detailed 2D-3D Joint Representation for Human-Object Interaction](https://mlsanity.com/paper/2004.08154/) (2020)
- [Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets](https://mlsanity.com/paper/2007.05405/) (2020)
- [Temporal Memory Relation Network for Workflow Recognition from Surgical Video](https://mlsanity.com/paper/2103.16327/) (2021)
- [Multi-Label Classification with Label Graph Superimposing](https://mlsanity.com/paper/1911.09243/) (2019)
- [The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classification](https://mlsanity.com/paper/2002.04264/) (2020)
- [Learning to Detect](https://mlsanity.com/paper/1805.07631/) (2018)
- [Visual Semantic Role Labeling for Video Understanding](https://mlsanity.com/paper/2104.00990/) (2021)
- [Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark](https://mlsanity.com/paper/2109.14956/) (2021)

## Built on by (most cited first, 30 of 44)

- [Dissecting Self-Supervised Learning Methods for Surgical Computer Vision](https://mlsanity.com/paper/2207.00449/) (2022)
- [Latent Graph Representations for Critical View of Safety Assessment](https://mlsanity.com/paper/2212.04155/) (2022)
- [EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery](https://mlsanity.com/paper/2501.11347/) (2025)
- [Rendezvous in Time: An Attention-based Temporal Fusion approach for Surgical Triplet Recognition](https://mlsanity.com/paper/2211.16963/) (2022)
- [Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets](https://mlsanity.com/paper/2204.05235/) (2022)
- [CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detection](https://mlsanity.com/paper/2302.06294/) (2023)
- [Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions](https://mlsanity.com/paper/2307.09548/) (2023)
- [Biomedical image analysis competitions: The state of current participation practice](https://mlsanity.com/paper/2212.08568/) (2022)
- [GP-VLS: A general-purpose vision language model for surgery](https://mlsanity.com/paper/2407.19305/) (2024)
- [OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understanding](https://mlsanity.com/paper/2504.14692/) (2025)
- [Surgical Tattoos in Infrared: A Dataset for Quantifying Tissue Tracking and Mapping](https://mlsanity.com/paper/2309.16782/) (2023)
- [Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoning](https://mlsanity.com/paper/2312.06829/) (2023)
- [Comparison of fine-tuning strategies for transfer learning in medical image classification](https://mlsanity.com/paper/2406.10050/) (2024)
- [Self-distillation for surgical action recognition](https://mlsanity.com/paper/2303.12915/) (2023)
- [Data Augmentation for Surgical Scene Segmentation with Anatomy-Aware Diffusion Models](https://mlsanity.com/paper/2410.07753/) (2024)
- [VS-Assistant: Versatile Surgery Assistant on the Demand of Surgeons](https://mlsanity.com/paper/2405.08272/) (2024)
- [Objective Surgical Skills Assessment and Tool Localization: Results from the MICCAI 2021 SimSurgSkill Challenge](https://mlsanity.com/paper/2212.04448/) (2022)
- [The SAGES Critical View of Safety Challenge: A Global Benchmark for AI-Assisted Surgical Quality Assessment](https://mlsanity.com/paper/2509.17100/) (2025)
- [Hypergraph-Transformer (HGT) for Interactive Event Prediction in Laparoscopic and Robotic Surgery](https://mlsanity.com/paper/2402.01974/) (2024)
- [Surgical Triplet Recognition via Diffusion Model](https://mlsanity.com/paper/2406.13210/) (2024)
- [ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detection](https://mlsanity.com/paper/2506.01130/) (2025)
- [MEJO: MLLM-Engaged Surgical Triplet Recognition via Inter- and Intra-Task Joint Optimization](https://mlsanity.com/paper/2509.12893/) (2025)
- [MURPHY: Relations Matter in Surgical Workflow Analysis](https://mlsanity.com/paper/2212.12719/) (2022)
- [Decoding the Surgical Scene: A Scoping Review of Scene Graphs in Surgery](https://mlsanity.com/paper/2509.20941/) (2025)
- [Robotic Scene Segmentation with Memory Network for Runtime Surgical Context Inference](https://mlsanity.com/paper/2308.12789/) (2023)
- [Surg-R1: A Hierarchical Reasoning Foundation Model for Scalable and Interpretable Surgical Decision Support with Multi-Center Clinical Validation](https://mlsanity.com/paper/2603.12430/) (2026)
- [DARIL: When Imitation Learning outperforms Reinforcement Learning in Surgical Action Planning](https://mlsanity.com/paper/2507.05011/) (2025)
- [SurgOnAir: Hierarchy-Aware Real-Time Surgical Video Commentary](https://mlsanity.com/paper/2605.21132/) (2026)
- [Multi-Stage Boundary-Aware Transformer Network for Action Segmentation in Untrimmed Surgical Videos](https://mlsanity.com/paper/2504.18756/) (2025)
- [Surgical Foundation Model Leveraging Compression and Entropy Maximization for Image-Guided Surgical Assistance](https://mlsanity.com/paper/2506.01980/) (2025)

## Abstract (arXiv)

Context-aware decision support in the operating room can foster surgical safety and efficiency by leveraging real-time feedback from surgical workflow analysis. Most existing works recognize surgical activities at a coarse-grained level, such as phases, steps or events, leaving out fine-grained interaction details about the surgical activity; yet those are needed for more helpful AI assistance in the operating room. Recognizing surgical actions as triplets of <instrument, verb, target> combination delivers comprehensive details about the activities taking place in surgical videos. This paper presents CholecTriplet2021: an endoscopic vision challenge organized at MICCAI 2021 for the recognition of surgical action triplets in laparoscopic videos. The challenge granted private access to the large-scale CholecT50 dataset, which is annotated with action triplet information. In this paper, we present the challenge setup and assessment of the state-of-the-art deep learning methods proposed by the participants during the challenge. A total of 4 baseline methods from the challenge organizers and 19 new deep learning algorithms by competing teams are presented to recognize surgical action triplets directly from surgical videos, achieving mean average precision (mAP) ranging from 4.2% to 38.1%. This study also analyzes the significance of the results obtained by the presented approaches, performs a thorough methodological comparison between them, in-depth result analysis, and proposes a novel ensemble method for enhanced recognition. Our analysis shows that surgical workflow analysis is not yet solved, and also highlights interesting directions for future research on fine-grained surgical activity recognition which is of utmost importance for the development of AI in surgery.