# Revisiting Distillation for Continual Learning on Visual Question Localized-Answering in Robotic Surgery

arXiv: https://arxiv.org/abs/2307.12045 · HTML: https://mlsanity.com/paper/2307.12045/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 22 Jul 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019)
- [Incremental Learning Techniques for Semantic Segmentation](https://mlsanity.com/paper/1907.13372/) (2019)
- [2018 Robotic Scene Segmentation Challenge](https://mlsanity.com/paper/2001.11190/) (2020)
- [Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2203.06359/) (2022)
- [Medical Visual Question Answering: A Survey](https://mlsanity.com/paper/2111.10056/) (2021)
- [PyCIL: A Python Toolbox for Class-Incremental Learning](https://mlsanity.com/paper/2112.12533/) (2021)
- [CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks](https://mlsanity.com/paper/2206.09059/) (2022)
- [Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformer](https://mlsanity.com/paper/2206.11053/) (2022)
- [Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks](https://mlsanity.com/paper/1802.08774/) (2018)
- [Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery](https://mlsanity.com/paper/2305.11692/) (2023)
- [The TUM LapChole dataset for the M2CAI 2016 workflow challenge](https://mlsanity.com/paper/1610.09278/) (2016)
- [Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task](https://mlsanity.com/paper/2208.12037/) (2022)
- [Learning and Reasoning with the Graph Structure Representation in Robotic Surgery](https://mlsanity.com/paper/2007.03357/) (2020)
- [LifeLonger: A Benchmark for Continual Disease Classification](https://mlsanity.com/paper/2204.05737/) (2022)
- [Self-Calibrated Efficient Transformer for Lightweight Super-Resolution](https://mlsanity.com/paper/2204.08913/) (2022)
- [Image Quality-aware Diagnosis via Meta-knowledge Co-embedding](https://mlsanity.com/paper/2303.15038/) (2023)
- [The Influence of Age and Gender Information on the Diagnosis of Diabetic Retinopathy: Based on Neural Networks](https://mlsanity.com/paper/2108.03026/) (2021)

## Built on by (most cited first, 15 of 15)

- [Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery](https://mlsanity.com/paper/2405.10948/) (2024)
- [Recent Advances of Multimodal Continual Learning: A Comprehensive Survey](https://mlsanity.com/paper/2410.05352/) (2024)
- [Continual Learning in Medical Image Analysis: A Comprehensive Review of Recent Advancements and Future Prospects](https://mlsanity.com/paper/2312.17004/) (2023)
- [Rethinking Exemplars for Continual Semantic Segmentation in Endoscopy Scenes: Entropy-based Mini-Batch Pseudo-Replay](https://mlsanity.com/paper/2308.14100/) (2023)
- [Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgery](https://mlsanity.com/paper/2408.04958/) (2024)
- [ORacle: Large Vision-Language Models for Knowledge-Guided Holistic OR Domain Modeling](https://mlsanity.com/paper/2404.07031/) (2024)
- [Continual Learning for Generative AI: From LLMs to MLLMs and Beyond](https://mlsanity.com/paper/2506.13045/) (2025)
- [A Survey of Medical Vision-and-Language Applications and Their Techniques](https://mlsanity.com/paper/2411.12195/) (2024)
- [EndoOOD: Uncertainty-aware Out-of-distribution Detection in Capsule Endoscopy Diagnosis](https://mlsanity.com/paper/2402.11476/) (2024)
- [OSSAR: Towards Open-Set Surgical Activity Recognition in Robot-assisted Surgery](https://mlsanity.com/paper/2402.06985/) (2024)
- [VLM-Assisted Continual learning for Visual Question Answering in Self-Driving](https://mlsanity.com/paper/2502.00843/) (2025)
- [Dynamic Robot-Assisted Surgery with Hierarchical Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2508.01713/) (2025)
- [Can DeepSeek Reason Like a Surgeon? An Empirical Evaluation for Vision-Language Understanding in Robotic-Assisted Surgery](https://mlsanity.com/paper/2503.23130/) (2025)
- [MacVQA: Adaptive Memory Allocation and Global Noise Filtering for Continual Visual Question Answering](https://mlsanity.com/paper/2601.01926/) (2026)
- [Bi-CRCL: Bidirectional Conservative-Radical Complementary Learning with Pre-trained Foundation Models for Class-incremental Medical Image Analysis](https://mlsanity.com/paper/2603.23729/) (2026)

## Abstract (arXiv)

The visual-question localized-answering (VQLA) system can serve as a knowledgeable assistant in surgical education. Except for providing text-based answers, the VQLA system can highlight the interested region for better surgical scene understanding. However, deep neural networks (DNNs) suffer from catastrophic forgetting when learning new knowledge. Specifically, when DNNs learn on incremental classes or tasks, their performance on old tasks drops dramatically. Furthermore, due to medical data privacy and licensing issues, it is often difficult to access old data when updating continual learning (CL) models. Therefore, we develop a non-exemplar continual surgical VQLA framework, to explore and balance the rigidity-plasticity trade-off of DNNs in a sequential learning paradigm. We revisit the distillation loss in CL tasks, and propose rigidity-plasticity-aware distillation (RP-Dist) and self-calibrated heterogeneous distillation (SH-Dist) to preserve the old knowledge. The weight aligning (WA) technique is also integrated to adjust the weight bias between old and new tasks. We further establish a CL framework on three public surgical datasets in the context of surgical settings that consist of overlapping classes between old and new surgical VQLA tasks. With extensive experiments, we demonstrate that our proposed method excellently reconciles learning and forgetting on the continual surgical VQLA over conventional CL methods. Our code is publicly accessible.