mlsanityData to 7 Oct 2026

Medical Vision Language Models as Policies for Robotic Surgery

arXiv 2510.06064 ↗First posted 7 Oct 2025cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-based Proximal Policy Optimization (PPO) struggles with visual observation-based robotic laparoscopic surgical tasks due to the high-dimensional nature of visual input, the sparsity of rewards in surgical environments, and the difficulty of extracting task-relevant features from raw visual data. We introduce a simple approach integrating MedFlamingo, a medical domain-specific Vision-Language Model, with PPO. Our method is evaluated on five diverse laparoscopic surgery task environments in LapGym, using only endoscopic visual observations. MedFlamingo PPO outperforms and converges faster compared to both standard vision-based PPO and OpenFlamingo PPO baselines, achieving task success rates exceeding 70% across all environments, with improvements ranging from 66.67% to 1114.29% compared to baseline. By processing task observations and instructions once per episode to generate high-level planning tokens, our method efficiently combines medical expertise with real-time visual feedback. Our results highlight the value of specialized medical knowledge in robotic surgical planning and decision-making.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 26 references that are in the map. First posted on arXiv 7 Oct 2025.

Builds on

17 of 26 references.

References · 17

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  10. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  11. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  12. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
5 more
  1. 2018Benchmarking Reinforcement Learning Algorithms on Real-World Robotscited by 64
  2. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  3. 2018Global overview of Imitation Learningcited by 16
  4. 2023LapGym -- An Open Source Framework for Reinforcement Learning in Robot-Assisted Laparoscopic Surgerycited by 12
  5. 2024Admittance Control for Adaptive Remote Center of Motion in Robotic Laparoscopic Surgerycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Systematic Evaluation of Large Vision-Language Models for Surgical Artificial Intelligence
  2. 2025Challenging Vision-Language Models with Surgical Data: A New Dataset and Broad Benchmarking Study
  3. 2026SurgVLA-Bench: Towards Evaluating Vision-Language-Action Models for Laparoscopic Surgical Robotics
  4. 2025SurgXBench: Explainable Vision-Language Model Benchmark for Surgery
  5. 2024GP-VLS: A general-purpose vision language model for surgery
  6. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence
  7. 2026SurGo-R1: Benchmarking and Modeling Contextual Reasoning for Operative Zone in Surgical Video
  8. 2025EndoVLA: Dual-Phase Vision-Language-Action Model for Autonomous Tracking in Endoscopy