# Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generation

arXiv: https://arxiv.org/abs/2306.02651 · HTML: https://mlsanity.com/paper/2306.02651/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Graph Neural Network for Traffic Forecasting: A Survey](https://mlsanity.com/paper/2101.11174/) (2021)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs](https://mlsanity.com/paper/1803.08035/) (2018)
- [Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation](https://mlsanity.com/paper/1805.08298/) (2018)
- [2018 Robotic Scene Segmentation Challenge](https://mlsanity.com/paper/2001.11190/) (2020)
- [TPH-YOLOv5: Improved YOLOv5 Based on Transformer Prediction Head for Object Detection on Drone-captured Scenarios](https://mlsanity.com/paper/2108.11539/) (2021)
- [Deep Reasoning with Knowledge Graph for Social Relationship Understanding](https://mlsanity.com/paper/1807.00504/) (2018)
- [Exploiting Semantics in Neural Machine Translation with Graph Convolutional Networks](https://mlsanity.com/paper/1804.08313/) (2018)
- [Class-Incremental Domain Adaptation with Smoothing and Calibration for Surgical Report Generation](https://mlsanity.com/paper/2107.11091/) (2021)
- [Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery](https://mlsanity.com/paper/2011.01619/) (2020)
- [Learning Domain Adaptation with Model Calibration for Surgical Report Generation in Robotic Surgery](https://mlsanity.com/paper/2103.17120/) (2021)
- [Rethinking Surgical Captioning: End-to-End Window-Based MLP Transformer Using Patches](https://mlsanity.com/paper/2207.00113/) (2022)

## Built on by (most cited first, 14 of 14)

- [RainMamba: Enhanced Locality Learning with State Space Models for Video Deraining](https://mlsanity.com/paper/2407.21773/) (2024)
- [Multimodal Large Language Models for Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2504.21051/) (2025)
- [HFGS: 4D Gaussian Splatting with Emphasis on Spatial and Temporal High-Frequency Components for Endoscopic Scene Reconstruction](https://mlsanity.com/paper/2405.17872/) (2024)
- [Deep Learning for Surgical Instrument Recognition and Segmentation in Robotic-Assisted Surgeries: A Systematic Review](https://mlsanity.com/paper/2410.07269/) (2024)
- [Semi-Supervised Video Desnowing Network via Temporal Decoupling Experts and Distribution-Driven Contrastive Regularization](https://mlsanity.com/paper/2410.07901/) (2024)
- [Language-Driven Interactive Shadow Detection](https://mlsanity.com/paper/2408.08543/) (2024)
- [Timeline and Boundary Guided Diffusion Network for Video Shadow Detection](https://mlsanity.com/paper/2408.11785/) (2024)
- [Non-Invasive to Invasive: Enhancing FFA Synthesis from CFP with a Benchmark Dataset and a Novel Network](https://mlsanity.com/paper/2410.14965/) (2024)
- [Decoding the Surgical Scene: A Scoping Review of Scene Graphs in Surgery](https://mlsanity.com/paper/2509.20941/) (2025)
- [CoC: Chain-of-Cancer based on Cross-Modal Autoregressive Traction for Survival Prediction](https://mlsanity.com/paper/2506.15696/) (2025)
- [More than Segmentation: Benchmarking SAM 3 for Segmentation, 3D Perception, and Reconstruction in Robotic Surgery](https://mlsanity.com/paper/2512.07596/) (2025)
- [Video-Instrument Synergistic Network for Referring Video Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/2308.09475/) (2023)
- [Can DeepSeek Reason Like a Surgeon? An Empirical Evaluation for Vision-Language Understanding in Robotic-Assisted Surgery](https://mlsanity.com/paper/2503.23130/) (2025)
- [SAGE-OR: Semi-supervised Adaptive Scene Graph Generation for Operating Rooms](https://mlsanity.com/paper/2608.15336/) (2026)

## Abstract (arXiv)

For robot-assisted surgery, an accurate surgical report reflects clinical operations during surgery and helps document entry tasks, post-operative analysis and follow-up treatment. It is a challenging task due to many complex and diverse interactions between instruments and tissues in the surgical scene. Although existing surgical report generation methods based on deep learning have achieved large success, they often ignore the interactive relation between tissues and instrumental tools, thereby degrading the report generation performance. This paper presents a neural network to boost surgical report generation by explicitly exploring the interactive relation between tissues and surgical instruments. We validate the effectiveness of our method on a widely-used robotic surgery benchmark dataset, and experimental results show that our network can significantly outperform existing state-of-the-art surgical report generation methods (e.g., 7.48% and 5.43% higher for BLEU-1 and ROUGE).