mlsanityData to 7 Oct 2026

A vision-language model and platform for temporally mapping surgery from video

arXiv 2603.22583 ↗First posted 23 Mar 2026cs.CV cs.ROcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Mapping surgery is fundamental to developing operative guidelines and enabling autonomous robotic surgery. Recent advances in artificial intelligence (AI) have shown promise in mapping the behaviour of surgeons from videos, yet current models remain narrow in scope, capturing limited behavioural components within single procedures, and offer limited translational value, as they remain inaccessible to practising surgeons. Here we introduce Halsted, a vision-language model trained on the Halsted Surgical Atlas (HSA), one of the most comprehensive annotated video libraries grown through an iterative self-labelling framework and encompassing over 650,000 videos across eight surgical specialties. To facilitate benchmarking, we publicly release HSA-27k, a subset of the Halsted Surgical Atlas. Halsted surpasses previous state-of-the-art models in mapping surgical activity while offering greater comprehensiveness and computational efficiency. To bridge the longstanding translational gap of surgical AI, we develop the Halsted web platform (https://halstedhealth.ai/) to provide surgeons anywhere in the world with the previously-unavailable capability of automatically mapping their own procedures within minutes. By standardizing unstructured surgical video data and making these capabilities directly accessible to surgeons, our work brings surgical AI closer to clinical deployment and helps pave the way toward autonomous robotic surgery.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 47 references that are in the map. First posted on arXiv 23 Mar 2026.

Builds on

19 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretraining23 Nov

Other references · 18

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  5. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  6. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  7. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  8. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  9. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  10. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  11. 2023Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lecturescited by 68
  12. 2023SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challengecited by 54
6 more
  1. 2022CholecTriplet2021: A benchmark challenge for surgical action triplet recognitioncited by 44
  2. 2017Video and Accelerometer-Based Motion Analysis for Automated Surgical Skills Assessmentcited by 28
  3. 2024General surgery vision transformer: A video pre-trained foundation model for general surgerycited by 25
  4. 2025Scaling up self-supervised learning for improved surgical foundation modelscited by 25
  5. 2025Large-scale Self-supervised Video Foundation Model for Intelligent Surgerycited by 10
  6. 2025SurgBench: A Unified Large-Scale Benchmark for Surgical Video Analysiscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Systematic Evaluation of Large Vision-Language Models for Surgical Artificial Intelligence
  2. 2026Surgical Video Generation From Diffusion to World Models: A Survey
  3. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence
  4. 2025SurgXBench: Explainable Vision-Language Model Benchmark for Surgery
  5. 2026A Comparative Study in Surgical AI: Potential and Limitations of Data, Compute, and Scaling
  6. 2026An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Models
  7. 2025Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling
  8. 2026Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planning