# Efficient Frame Extraction: A Novel Approach Through Frame Similarity and Surgical Tool Tracking for Video Segmentation arXiv: https://arxiv.org/abs/2501.11153 ยท HTML: https://mlsanity.com/paper/2501.11153/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 80 references that are in the map. - First posted on arXiv 19 Jan 2025. ## Builds on (most cited first, 28 of 28) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021) - [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015) - [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013) - [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021) - [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020) - [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014) - [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016) - [FLatten Transformer: Vision Transformer using Focused Linear Attention](https://mlsanity.com/paper/2308.00442/) (2023) - [TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks](https://mlsanity.com/paper/2003.10751/) (2020) - [Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis](https://mlsanity.com/paper/1907.06099/) (2019) - [LoViT: Long Video Transformer for Surgical Phase Recognition](https://mlsanity.com/paper/2305.08989/) (2023) - [Dynamic Scene Graph Representation for Surgical Video](https://mlsanity.com/paper/2309.14538/) (2023) - [A spatio-temporal network for video semantic segmentation in surgical videos](https://mlsanity.com/paper/2306.11052/) (2023) - [Metrics Matter in Surgical Phase Recognition](https://mlsanity.com/paper/2305.13961/) (2023) - [3DFCNN: Real-Time Action Recognition using 3D Deep Neural Networks with Raw Depth Information](https://mlsanity.com/paper/2006.07743/) (2020) - [SurgPLAN: Surgical Phase Localization Network for Phase Recognition](https://mlsanity.com/paper/2311.09965/) (2023) - [Recurrent Fully Convolutional Networks for Video Segmentation](https://mlsanity.com/paper/1606.00487/) (2016) - [Alzheimer's Disease Diagnostics by a Deeply Supervised Adaptable 3D Convolutional Network](https://mlsanity.com/paper/1607.00556/) (2016) - [Preserving Privacy in Surgical Video Analysis Using Artificial Intelligence: A Deep Learning Classifier to Identify Out-of-Body Scenes in Endoscopic Videos](https://mlsanity.com/paper/2301.07053/) (2023) - [M2CAI Workflow Challenge: Convolutional Neural Networks with Time Smoothing and Hidden Markov Model for Video Frames Classification](https://mlsanity.com/paper/1610.05541/) (2016) - [Video Action Recognition Collaborative Learning with Dynamics via PSO-ConvNet Transformer](https://mlsanity.com/paper/2302.09187/) (2023) - [Advanced Capsule Networks via Context Awareness](https://mlsanity.com/paper/1903.07497/) (2019) - [An Improvement for Capsule Networks using Depthwise Separable Convolution](https://mlsanity.com/paper/2007.15167/) (2020) - [Evaluating Model Performance with Hard-Swish Activation Function Adjustments](https://mlsanity.com/paper/2410.06879/) (2024) - [PSO-Convolutional Neural Networks with Heterogeneous Learning Rate](https://mlsanity.com/paper/2205.10456/) (2022) - [Machine Learning-Based Automated Assessment of Intracorporeal Suturing in Laparoscopic Fundoplication](https://mlsanity.com/paper/2412.16195/) (2024) ## Built on by (most cited first, 2 of 2) - [Structured Proxy Features for Multimodal NSCLC Survival Prediction from Pretreatment CT](https://mlsanity.com/paper/2608.00446/) (2026) - [Extended KAFR: A kinematic-adaptive paradigm for the efficient analysis of surgical video](https://mlsanity.com/paper/2608.01058/) (2026) ## Abstract (arXiv) The interest in leveraging Artificial Intelligence (AI) for surgical procedures to automate analysis has witnessed a significant surge in recent years. One of the primary tools for recording surgical procedures and conducting subsequent analyses, such as performance assessment, is through videos. However, these operative videos tend to be notably lengthy compared to other fields, spanning from thirty minutes to several hours, which poses a challenge for AI models to effectively learn from them. Despite this challenge, the foreseeable increase in the volume of such videos in the near future necessitates the development and implementation of innovative techniques to tackle this issue effectively. In this article, we propose a novel technique called Kinematics Adaptive Frame Recognition (KAFR) that can efficiently eliminate redundant frames to reduce dataset size and computation time while retaining useful frames to improve accuracy. Specifically, we compute the similarity between consecutive frames by tracking the movement of surgical tools. Our approach follows these steps: i) Tracking phase: a YOLOv8 model is utilized to detect tools presented in the scene, ii) Similarity phase: Similarities between consecutive frames are computed by estimating variation in the spatial positions and velocities of the tools, iii) Classification phase: An X3D CNN is trained to classify segmentation. We evaluate the effectiveness of our approach by analyzing datasets obtained through retrospective reviews of cases at two referral centers. The newly annotated Gastrojejunostomy (GJ) dataset covers procedures performed between 2017 and 2021, while the previously annotated Pancreaticojejunostomy (PJ) dataset spans from 2011 to 2022 at the same centers.