# Dynamic Robot-Assisted Surgery with Hierarchical Class-Incremental Semantic Segmentation

arXiv: https://arxiv.org/abs/2508.01713 · HTML: https://mlsanity.com/paper/2508.01713/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 3 Aug 2025.

## Builds on (most cited first, 27 of 27)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Hyperbolic Neural Networks](https://mlsanity.com/paper/1805.09112/) (2018)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](https://mlsanity.com/paper/2011.11390/) (2020)
- [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019)
- [2018 Robotic Scene Segmentation Challenge](https://mlsanity.com/paper/2001.11190/) (2020)
- [Representation Compensation Networks for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.05402/) (2022)
- [Continual Learning for Abdominal Multi-Organ and Tumor Segmentation](https://mlsanity.com/paper/2306.00988/) (2023)
- [Decomposed Knowledge Distillation for Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2210.05941/) (2022)
- [MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments](https://mlsanity.com/paper/2503.02579/) (2025)
- [Learn the new, keep the old: Extending pretrained models with new anatomy and images](https://mlsanity.com/paper/1806.00265/) (2018)
- [Extending Pretrained Segmentation Networks with Additional Anatomical Structures](https://mlsanity.com/paper/1811.04634/) (2018)
- [Attribution-aware Weight Transfer: A Warm-Start Initialization for Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2210.07207/) (2022)
- [Continual Segment: Towards a Single, Unified and Accessible Continual Segmentation Model of 143 Whole-body Organs in CT Scans](https://mlsanity.com/paper/2302.00162/) (2023)
- [Learning Incrementally to Segment Multiple Organs in a CT Image](https://mlsanity.com/paper/2203.02100/) (2022)
- [Revisiting Distillation for Continual Learning on Visual Question Localized-Answering in Robotic Surgery](https://mlsanity.com/paper/2307.12045/) (2023)
- [Syn-Mediverse: A Multimodal Synthetic Dataset for Intelligent Scene Understanding of Healthcare Facilities](https://mlsanity.com/paper/2308.03193/) (2023)
- [CoVIO: Online Continual Learning for Visual-Inertial Odometry](https://mlsanity.com/paper/2303.10149/) (2023)
- [Incremental Learning for Heterogeneous Structure Segmentation in Brain Tumor MRI](https://mlsanity.com/paper/2305.19404/) (2023)
- [Continual Learning in Medical Imaging: A Survey and Practical Analysis](https://mlsanity.com/paper/2405.13482/) (2024)
- [Taxonomy-Aware Continual Semantic Segmentation in Hyperbolic Spaces for Open-World Perception](https://mlsanity.com/paper/2407.18145/) (2024)
- [L3DMC: Lifelong Learning using Distillation via Mixed-Curvature Space](https://mlsanity.com/paper/2307.16459/) (2023)
- [Privacy-Preserving Synthetic Continual Semantic Segmentation for Robotic Surgery](https://mlsanity.com/paper/2402.05860/) (2024)
- [Segmentation of Multiple Sclerosis Lesions across Hospitals: Learn Continually or Train from Scratch?](https://mlsanity.com/paper/2210.15091/) (2022)
- [MultiCo3D: Multi-Label Voxel Contrast for One-Shot Incremental Segmentation of 3D Neuroimages](https://mlsanity.com/paper/2503.06598/) (2025)

## Built on by (most cited first, 3 of 3)

- [ForecastOcc: Vision-based Semantic Occupancy Forecasting](https://mlsanity.com/paper/2602.08006/) (2026)
- [Hyp2Former: Hierarchy-Aware Hyperbolic Embeddings for Open-Set Panoptic Segmentation](https://mlsanity.com/paper/2605.02580/) (2026)
- [Human-Centered Reflections on Care Robots: A Comparative Study of Caregiver Perspectives](https://mlsanity.com/paper/2608.02411/) (2026)

## Abstract (arXiv)

Robot-assisted surgeries rely on accurate and real-time scene understanding to safely guide surgical instruments. However, segmentation models trained on static datasets face key limitations when deployed in these dynamic and evolving surgical environments. Class-incremental semantic segmentation (CISS) allows models to continually adapt to new classes while avoiding catastrophic forgetting of prior knowledge, without training on previous data. In this work, we build upon the recently introduced Taxonomy-Oriented Poincar'e-regularized Incremental Class Segmentation (TOPICS) approach and propose an enhanced variant, termed TOPICS+, specifically tailored for robust segmentation of surgical scenes. Concretely, we incorporate the Dice loss into the hierarchical loss formulation to handle strong class imbalances, introduce hierarchical pseudo-labeling, and design tailored label taxonomies for robotic surgery environments. We also propose six novel CISS benchmarks designed for robotic surgery environments including multiple incremental steps and several semantic categories to emulate realistic class-incremental settings in surgical environments. In addition, we introduce a refined set of labels with more than 144 classes on the Syn-Mediverse synthetic dataset, hosted online as an evaluation benchmark. We make the code and trained models publicly available at http://topics.cs.uni-freiburg.de.