mlsanityData to 14 Oct 2026

BronchoCopilot: Towards Autonomous Robotic Bronchoscopy via Multimodal Reinforcement Learning

arXiv 2403.01483 ↗First posted 3 Mar 2024cs.ROcited by 2

On the map: Deep reinforcement learning methods → RL for clinical treatment and surgery

Abstract (arXiv)

Bronchoscopy plays a significant role in the early diagnosis and treatment of lung diseases. This process demands physicians to maneuver the flexible endoscope for reaching distal lesions, particularly requiring substantial expertise when examining the airways of the upper lung lobe. With the development of artificial intelligence and robotics, reinforcement learning (RL) method has been applied to the manipulation of interventional surgical robots. However, unlike human physicians who utilize multimodal information, most of the current RL methods rely on a single modality, limiting their performance. In this paper, we propose BronchoCopilot, a multimodal RL agent designed to acquire manipulation skills for autonomous bronchoscopy. BronchoCopilot specifically integrates images from the bronchoscope camera and estimated robot poses, aiming for a higher success rate within challenging airway environment. We employ auxiliary reconstruction tasks to compress multimodal data and utilize attention mechanisms to achieve an efficient latent representation of this data, serving as input for the RL module. This framework adopts a stepwise training and fine-tuning approach to mitigate the challenges of training difficulty. Our evaluation in the realistic simulation environment reveals that BronchoCopilot, by effectively harnessing multimodal information, attains a success rate of approximately 90% in fifth generation airways with consistent movements. Additionally, it demonstrates a robust capacity to adapt to diverse cases.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL for clinical treatment and surgery, one of 15 directions in Deep reinforcement learning methods. Builds on 21 of its 49 references that are in the map. First posted on arXiv 3 Mar 2024.

Builds on

21 of 49 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  6. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 153
  7. 2018Recent Advances in Autoencoder-Based Representation Learningcited by 133
  8. 2019Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Taskscited by 84
  9. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 81
  10. 2020Alleviating Class-wise Gradient Imbalance for Pulmonary Airway Segmentationcited by 35
  11. 2019Reinforcement Learning with Attention that Works: A Self-Supervised Approachcited by 19
  12. 2017Learning End-to-end Multimodal Sensor Policies for Autonomous Navigationcited by 18
9 more
  1. 2017Robot Autonomy for Surgerycited by 17
  2. 2019Embodied Multimodal Multitask Learningcited by 13
  3. 2019Autonomous Driving in the Lung using Deep Learning for Localizationcited by 13
  4. 2021Deep reinforcement learning for guidewire navigation in coronary artery phantomcited by 10
  5. 2023Autonomous Navigation for Robot-assisted Intraluminal and Endovascular Procedures: A Systematic Reviewcited by 8
  6. 2022Colonoscopy Navigation using End-to-End Deep Visuomotor Control: A User Studycited by 6
  7. 2017Crossmodal Attentive Skill Learnercited by 4
  8. 2024Design and Visual Servoing Control of a Hybrid Dual-Segment Flexible Neurosurgical Robot for Intraventricular Biopsycited by 2
  9. 2023Effective Multimodal Reinforcement Learning with Modality Alignment and Importance Enhancementcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026BronchoLumen: Analysis of recent YOLO-based architectures for real-time bronchial orifice detection in video bronchoscopycited by 0
  2. 2026StenoVLA-3D: 3D-Aware Reasoning VLA for Navigation Through Gastrointestinal Stenosescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Long-Short Term Agents for Pure-Vision Bronchoscopy Robotic Autonomy
  2. 2026Anatomy-Aware Prediction of Bronchoscopic Accessibility from 3D CT
  3. 2024Autonomous Path Planning for Intercostal Robotic Ultrasound Imaging Using Reinforcement Learning
  4. 2026BiliVLA: Scene-Aware Vision-Language-Action Model with Reinforcement Learning for Autonomous Biliary Endoscopic Navigation
  5. 2026AdaFuse: Adaptive Multimodal Fusion for Lung Cancer Risk Prediction via Reinforcement Learning
  6. 2026BREATH-VL: Vision-Language-Guided 6-DoF Bronchoscopy Localization via Semantic-Geometric Fusion
  7. 2023CASOG: Conservative Actor-critic with SmOoth Gradient for Skill Learning in Robot-Assisted Intervention
  8. 2026Geometry-Aware Camera Localization for Bronchoscopy