# BronchoCopilot: Towards Autonomous Robotic Bronchoscopy via Multimodal Reinforcement Learning

arXiv: https://arxiv.org/abs/2403.01483 · HTML: https://mlsanity.com/paper/2403.01483/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL for clinical treatment and surgery, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 49 references that are in the map.
- First posted on arXiv 3 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [RL for clinical treatment and surgery](https://mlsanity.com/deeprl/rl-for-clinical-treatment-and-surgery/)

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [Recent Advances in Autoencoder-Based Representation Learning](https://mlsanity.com/paper/1812.05069/) (2018)
- [Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1907.13098/) (2019)
- [Mapping Instructions and Visual Observations to Actions with Reinforcement Learning](https://mlsanity.com/paper/1704.08795/) (2017)
- [Alleviating Class-wise Gradient Imbalance for Pulmonary Airway Segmentation](https://mlsanity.com/paper/2011.11952/) (2020)
- [Reinforcement Learning with Attention that Works: A Self-Supervised Approach](https://mlsanity.com/paper/1904.03367/) (2019)
- [Learning End-to-end Multimodal Sensor Policies for Autonomous Navigation](https://mlsanity.com/paper/1705.10422/) (2017)
- [Robot Autonomy for Surgery](https://mlsanity.com/paper/1707.03080/) (2017)
- [Embodied Multimodal Multitask Learning](https://mlsanity.com/paper/1902.01385/) (2019)
- [Autonomous Driving in the Lung using Deep Learning for Localization](https://mlsanity.com/paper/1907.08136/) (2019)
- [Deep reinforcement learning for guidewire navigation in coronary artery phantom](https://mlsanity.com/paper/2110.01840/) (2021)
- [Autonomous Navigation for Robot-assisted Intraluminal and Endovascular Procedures: A Systematic Review](https://mlsanity.com/paper/2305.04027/) (2023)
- [Colonoscopy Navigation using End-to-End Deep Visuomotor Control: A User Study](https://mlsanity.com/paper/2206.15086/) (2022)
- [Crossmodal Attentive Skill Learner](https://mlsanity.com/paper/1711.10314/) (2017)
- [Design and Visual Servoing Control of a Hybrid Dual-Segment Flexible Neurosurgical Robot for Intraventricular Biopsy](https://mlsanity.com/paper/2402.09679/) (2024)
- [Effective Multimodal Reinforcement Learning with Modality Alignment and Importance Enhancement](https://mlsanity.com/paper/2302.09318/) (2023)

## Built on by (most cited first, 2 of 2)

- [BronchoLumen: Analysis of recent YOLO-based architectures for real-time bronchial orifice detection in video bronchoscopy](https://mlsanity.com/paper/2605.11748/) (2026)
- [StenoVLA-3D: 3D-Aware Reasoning VLA for Navigation Through Gastrointestinal Stenoses](https://mlsanity.com/paper/2609.24187/) (2026)

## Abstract (arXiv)

Bronchoscopy plays a significant role in the early diagnosis and treatment of lung diseases. This process demands physicians to maneuver the flexible endoscope for reaching distal lesions, particularly requiring substantial expertise when examining the airways of the upper lung lobe. With the development of artificial intelligence and robotics, reinforcement learning (RL) method has been applied to the manipulation of interventional surgical robots. However, unlike human physicians who utilize multimodal information, most of the current RL methods rely on a single modality, limiting their performance. In this paper, we propose BronchoCopilot, a multimodal RL agent designed to acquire manipulation skills for autonomous bronchoscopy. BronchoCopilot specifically integrates images from the bronchoscope camera and estimated robot poses, aiming for a higher success rate within challenging airway environment. We employ auxiliary reconstruction tasks to compress multimodal data and utilize attention mechanisms to achieve an efficient latent representation of this data, serving as input for the RL module. This framework adopts a stepwise training and fine-tuning approach to mitigate the challenges of training difficulty. Our evaluation in the realistic simulation environment reveals that BronchoCopilot, by effectively harnessing multimodal information, attains a success rate of approximately 90% in fifth generation airways with consistent movements. Additionally, it demonstrates a robust capacity to adapt to diverse cases.