# BREATH-VL: Vision-Language-Guided 6-DoF Bronchoscopy Localization via Semantic-Geometric Fusion

arXiv: https://arxiv.org/abs/2601.03713 · HTML: https://mlsanity.com/paper/2601.03713/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 41 of its 57 references that are in the map.
- First posted on arXiv 7 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 30 of 41)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization](https://mlsanity.com/paper/2309.16020/) (2023)
- [ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2304.03047/) (2023)
- [Adaptive Keyframe Sampling for Long Video Understanding](https://mlsanity.com/paper/2502.21271/) (2025)
- [Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos using Depth Networks and Photometric Constraints](https://mlsanity.com/paper/2103.16525/) (2021)
- [Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registration](https://mlsanity.com/paper/2206.08903/) (2022)
- [Self-Supervised Monocular Depth and Ego-Motion Estimation in Endoscopy: Appearance Flow to the Rescue](https://mlsanity.com/paper/2112.08122/) (2021)
- [EndoMapper dataset of complete calibrated endoscopy procedures](https://mlsanity.com/paper/2204.14240/) (2022)
- [EndoGSLAM: Real-Time Dense Reconstruction and Tracking in Endoscopic Surgeries using Gaussian Splatting](https://mlsanity.com/paper/2403.15124/) (2024)
- [GeoReasoner: Geo-localization with Reasoning in Street Views using a Large Vision-Language Model](https://mlsanity.com/paper/2406.18572/) (2024)
- [Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoning](https://mlsanity.com/paper/2307.06166/) (2023)
- [AddressCLIP: Empowering Vision-Language Models for City-wide Image Address Localization](https://mlsanity.com/paper/2407.08156/) (2024)
- [SimCol3D -- 3D Reconstruction during Colonoscopy Challenge](https://mlsanity.com/paper/2307.11261/) (2023)
- [C3VDv2 -- Colonoscopy 3D video dataset with enhanced realism](https://mlsanity.com/paper/2506.24074/) (2025)
- [Keyframe-based monocular SLAM: design, survey, and future directions](https://mlsanity.com/paper/1607.00470/) (2016)
- [Autonomous Driving in the Lung using Deep Learning for Localization](https://mlsanity.com/paper/1907.08136/) (2019)
- [BronchoTrack: Airway Lumen Tracking for Branch-Level Bronchoscopic Localization](https://mlsanity.com/paper/2402.12763/) (2024)
- [EndoMamba: An Efficient Foundation Model for Endoscopic Videos via Hierarchical Pre-training](https://mlsanity.com/paper/2502.19090/) (2025)
- [EndoFlow-SLAM: Real-Time Endoscopic SLAM with Flow-Constrained Gaussian Splatting](https://mlsanity.com/paper/2506.21420/) (2025)
- [FiLM-Nav: Efficient and Generalizable Navigation via VLM Fine-tuning](https://mlsanity.com/paper/2509.16445/) (2025)
- [OffsetNet: Deep Learning for Localization in the Lung using Rendered Images](https://mlsanity.com/paper/1809.05645/) (2018)
- [BronchoPose: an analysis of data and model configuration for vision-based bronchoscopy pose estimation](https://mlsanity.com/paper/2204.11982/) (2022)

## Built on by (most cited first, 2 of 2)

- [Stop Holding Your Breath: CT-Informed Gaussian Splatting for Dynamic Bronchoscopy](https://mlsanity.com/paper/2604.28179/) (2026)
- [Geometry-Aware Camera Localization for Bronchoscopy](https://mlsanity.com/paper/2608.07116/) (2026)

## Abstract (arXiv)

Vision-language models (VLMs) have recently shown remarkable performance in navigation and localization tasks by leveraging large-scale pretraining for semantic understanding. However, applying VLMs to 6-DoF endoscopic camera localization presents several challenges: 1) the lack of large-scale, high-quality, densely annotated, and localization-oriented vision-language datasets in real-world medical settings; 2) limited capability for fine-grained pose regression; and 3) high computational latency when extracting temporal features from past frames. To address these issues, we first construct BREATH dataset, the largest in-vivo endoscopic localization dataset to date, collected in the complex human airway. Building on this dataset, we propose BREATH-VL, a hybrid framework that integrates semantic cues from VLMs with geometric information from vision-based registration methods for accurate 6-DoF pose estimation. Our motivation lies in the complementary strengths of both approaches: VLMs offer generalizable semantic understanding, while registration methods provide precise geometric alignment. To further enhance the VLM's ability to capture temporal context, we introduce a lightweight context-learning mechanism that encodes motion history as linguistic prompts, enabling efficient temporal reasoning without expensive video-level computation. Extensive experiments demonstrate that the vision-language module delivers robust semantic localization in challenging surgical scenes. Building on this, our BREATH-VL outperforms state-of-the-art vision-only localization methods in both accuracy and generalization, reducing translational error by 25.5% compared with the best-performing baseline, while achieving competitive computational latency.