mlsanityData to 7 Oct 2026

Chain-of-Adaptation: Surgical Vision-Language Adaptation with Reinforcement Learning

arXiv 2603.20116 ↗First posted 20 Mar 2026cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Conventional fine-tuning on domain-specific datasets can inadvertently alter a model's pretrained multimodal priors, leading to reduced generalization. To address this, we propose Chain-of-Adaptation (CoA), an adaptation framework designed to integrate domain knowledge while maintaining the model's inherent reasoning and perceptual capabilities. CoA introduces a structured reasoning format that enhances domain alignment without sacrificing general multimodal competence by reinforcement learning. Experiments on standard surgical benchmarks, under both in-distribution and out-of-distribution settings, demonstrate that CoA achieves higher accuracy, stronger generalization, and more stable behavior than supervised fine-tuning. Furthermore, ablation studies confirm that CoA effectively preserves the model's core visual-language abilities, providing a reliable pathway for domain specialization in VLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 39 references that are in the map. First posted on arXiv 20 Mar 2026.

Builds on

35 of 39 references.

References · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  12. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
23 more
  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  3. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  4. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  5. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  6. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  7. 2016EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videoscited by 366
  8. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  9. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  10. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 151
  11. 2022AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomycited by 85
  12. 2021Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmarkcited by 58
  13. 2020Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Roomcited by 47
  14. 2023Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgerycited by 44
  15. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  16. 2016The TUM LapChole dataset for the M2CAI 2016 workflow challengecited by 37
  17. 2025EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgerycited by 37
  18. 2024LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learningcited by 34
  19. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligencecited by 30
  20. 2025G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learningcited by 29
  21. 2021MIcro-Surgical Anastomose Workflow recognition challenge reportcited by 27
  22. 2023CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detectioncited by 22
  23. 2024GP-VLS: A general-purpose vision language model for surgerycited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforced Curriculum Pre-Alignment for Domain-Adaptive VLMs
  2. 2025Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learning
  3. 2026MMedExpert-R1: Strengthening Multimodal Medical Reasoning via Domain-Specific Adaptation and Clinical Guideline Reinforcement
  4. 2026Learning Domain Knowledge in Multimodal Large Language Models through Reinforcement Fine-Tuning
  5. 2026BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation
  6. 2026GRAFT: Grounded and Efficient Online Reinforcement Adaptation for Fine-Grained Robot Manipulation
  7. 2025Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Models
  8. 2026ClinCoT: Clinical-Aware Visual Chain-of-Thought for Medical Vision Language Models