mlsanityData to 7 Oct 2026

Multitask Learning in Minimally Invasive Surgical Vision: A Review

arXiv 2401.08256 ↗First posted 16 Jan 2024cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Minimally invasive surgery (MIS) has revolutionized many procedures and led to reduced recovery time and risk of patient injury. However, MIS poses additional complexity and burden on surgical teams. Data-driven surgical vision algorithms are thought to be key building blocks in the development of future MIS systems with improved autonomy. Recent advancements in machine learning and computer vision have led to successful applications in analyzing videos obtained from MIS with the promise of alleviating challenges in MIS videos. Surgical scene and action understanding encompasses multiple related tasks that, when solved individually, can be memory-intensive, inefficient, and fail to capture task relationships. Multitask learning (MTL), a learning paradigm that leverages information from multiple related tasks to improve performance and aid generalization, is well suited for fine-grained and high-level understanding of MIS data. This review provides a narrative overview of the current state-of-the-art MTL systems that leverage videos obtained from MIS. Beyond listing published approaches, we discuss the benefits and limitations of these MTL systems. Moreover, this manuscript presents an analysis of the literature for various application fields of MTL in MIS, including those with large models, highlighting notable trends, new directions of research, and developments.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 119 of its 185 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

119 of 185 references.

References · 119

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2017Mask R-CNNcited by 7,259
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  8. 2017Squeeze-and-Excitation Networkscited by 4,763
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  11. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  12. 2015Fast R-CNNcited by 2,930
38 more
  1. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  2. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  3. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  4. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  5. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  6. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  7. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2021Multiscale Vision Transformerscited by 896
  10. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 878
  11. 2017A Survey on Multi-Task Learningcited by 838
  12. 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 819
  13. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  14. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  15. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  16. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  17. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 515
  18. 2021Image Segmentation Using Text and Image Promptscited by 509
  19. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
  20. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  21. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  22. 2020U2-Net: Going Deeper with Nested U-Structure for Salient Object Detectioncited by 410
  23. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  24. 2016EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videoscited by 366
  25. 2014Deeply-Supervised Netscited by 364
  26. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  27. 2018PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsingcited by 224
  28. 2019Pareto Multi-Task Learningcited by 224
  29. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  30. 2017Multi-task Self-Supervised Visual Learningcited by 200
  31. 2016CNN-RNN: A Unified Framework for Multi-label Image Classificationcited by 189
  32. 2018Boundary loss for highly unbalanced segmentationcited by 187
  33. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  34. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 167
  35. 2020TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networkscited by 139
  36. 2021Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videoscited by 139
  37. 2019Neural-Guided RANSAC: Learning Where to Sample Model Hypothesescited by 129
  38. 2018NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reductioncited by 127

50 most cited of 119 All 119 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2025EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgerycited by 37
  2. 2024PitVis-2023 Challenge: Workflow Recognition in videos of Endoscopic Pituitary Surgerycited by 10
  3. 2025Surgical Video Understanding with Label Interpolationcited by 1
  4. 2026SurgFed: Language-guided Multi-Task Federated Learning for Surgical Video Understandingcited by 0
  5. 2026RoboSurg-VQA: A Multimodal Benchmark for Surgical Segmentation-Aware Visual Question Answeringcited by 0