# LLM-Assisted Multi-Teacher Continual Learning for Visual Question Answering in Robotic Surgery

arXiv: https://arxiv.org/abs/2402.16664 · HTML: https://mlsanity.com/paper/2402.16664/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- First posted on arXiv 26 Feb 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Built on by (most cited first, 20 of 20)

- [Large Language Models for Medicine: A Survey](https://mlsanity.com/paper/2405.13055/) (2024)
- [EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery](https://mlsanity.com/paper/2501.11347/) (2025)
- [A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directions](https://mlsanity.com/paper/2406.03712/) (2024)
- [SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence](https://mlsanity.com/paper/2506.02555/) (2025)
- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [Continual Learning for Generative AI: From LLMs to MLLMs and Beyond](https://mlsanity.com/paper/2506.13045/) (2025)
- [When Continue Learning Meets Multimodal Large Language Model: A Survey](https://mlsanity.com/paper/2503.01887/) (2025)
- [StyleMotif: Multi-Modal Motion Stylization using Style-Content Cross Fusion](https://mlsanity.com/paper/2503.21775/) (2025)
- [VLM-Assisted Continual learning for Visual Question Answering in Self-Driving](https://mlsanity.com/paper/2502.00843/) (2025)
- [Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learning](https://mlsanity.com/paper/2506.19469/) (2025)
- [Surgical-MambaLLM: Mamba2-enhanced Multimodal Large Language Model for VQLA in Robotic Surgery](https://mlsanity.com/paper/2509.16618/) (2025)
- [Prompt Customization for Continual Learning](https://mlsanity.com/paper/2404.18060/) (2024)
- [Analytic Subspace Routing: How Recursive Least Squares Works in Continual Learning of Large Language Model](https://mlsanity.com/paper/2503.13575/) (2025)
- [SurgSigma: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence](https://mlsanity.com/paper/2603.16822/) (2026)
- [Forging a Dynamic Memory: Retrieval-Guided Continual Learning for Generalist Medical Foundation Models](https://mlsanity.com/paper/2512.13072/) (2025)
- [Probing to Refine: Reinforcement Distillation of LLMs via Explanatory Inversion](https://mlsanity.com/paper/2603.19266/) (2026)
- [Rephrase and Contrast: Fine-Tuning Language Models for Enhanced Understanding of Communication and Computer Networks](https://mlsanity.com/paper/2409.19007/) (2024)
- [The Dual-use Dilemma in LLMs: Do Empowering Ethical Capacities Make a Degraded Utility?](https://mlsanity.com/paper/2501.13952/) (2025)
- [Continually Evolved Multimodal Foundation Models for Cancer Prognosis](https://mlsanity.com/paper/2501.18170/) (2025)
- [Pseudocode-Guided Structured Reasoning for Automating Reliable Inference in Vision-Language Models](https://mlsanity.com/paper/2605.19663/) (2026)

## Abstract (arXiv)

Visual question answering (VQA) is crucial for promoting surgical education. In practice, the needs of trainees are constantly evolving, such as learning more surgical types, adapting to different robots, and learning new surgical instruments and techniques for various surgeries. However, patient data privacy often restricts the availability of old data when updating the model, necessitating an exemplar-free continual learning (CL) setup. Prior CL studies overlooked two vital problems in the surgical domain: 1) large domain shifts from diverse surgical operations collected from multiple sources, and 2) severe data imbalance arising from the uneven presence of surgical instruments or activities. This paper proposes addressing these problems with a multimodal large language model (LLM) and an adaptive weight assignment methodology. We first develop a new multi-teacher CL framework that leverages a multimodal LLM as the additional teacher. The strong generalization ability of the LLM can bridge the knowledge gap when domain shifts and data imbalances occur. We then put forth a novel data processing method that transforms complex LLM embeddings into logits compatible with our CL framework. We further design an adaptive weight assignment approach that balances the generalization ability of the LLM and the domain expertise of the old CL model. Finally, to comprehensively test the effectiveness of our proposed method, we have also constructed two new surgical VQA datasets that are largely different from existing ones and could be valuable resources for future research. Extensive experimental results on the tested datasets demonstrate the superiority of our method to other advanced CL schemes.