# VesNet-RL: Simulation-based Reinforcement Learning for Real-World US Probe Navigation

arXiv: https://arxiv.org/abs/2205.06676 · HTML: https://mlsanity.com/paper/2205.06676/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 10 May 2022.

## Builds on (most cited first, 13 of 13)

- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning](https://mlsanity.com/paper/1803.09956/) (2018)
- [SonoNet: Real-Time Detection and Localisation of Fetal Standard Scan Planes in Freehand Ultrasound](https://mlsanity.com/paper/1612.05601/) (2016)
- [Towards MRI-Based Autonomous Robotic US Acquisitions: A First Feasibility Study](https://mlsanity.com/paper/1607.08371/) (2016)
- [Automatic Probe Movement Guidance for Freehand Obstetric Ultrasound](https://mlsanity.com/paper/2007.04480/) (2020)
- [Autonomous Robotic Screening of Tubular Structures based only on Real-Time Ultrasound Imaging Feedback](https://mlsanity.com/paper/2011.00099/) (2020)
- [Autonomous Navigation of an Ultrasound Probe Towards Standard Scan Planes with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.00718/) (2021)
- [Ultrasound-Guided Robotic Navigation with Deep Reinforcement Learning](https://mlsanity.com/paper/2003.13321/) (2020)
- [Deformation-Aware Robotic 3D Ultrasound](https://mlsanity.com/paper/2107.08411/) (2021)
- [A Learning-Driven Framework with Spatial Optimization For Surgical Suture Thread Reconstruction and Autonomous Grasping Under Multiple Topologies and Environmental Noises](https://mlsanity.com/paper/2007.00920/) (2020)

## Built on by (most cited first, 12 of 12)

- [Intelligent Robotic Sonographer: Mutual Information-based Disentangled Reward Learning from Few Demonstrations](https://mlsanity.com/paper/2307.03705/) (2023)
- [Towards Autonomous Atlas-based Ultrasound Acquisitions in Presence of Articulated Motion](https://mlsanity.com/paper/2208.05399/) (2022)
- [Motion Magnification in Robotic Sonography: Enabling Pulsation-Aware Artery Segmentation](https://mlsanity.com/paper/2307.03698/) (2023)
- [Goal-conditioned reinforcement learning for ultrasound navigation guidance](https://mlsanity.com/paper/2405.01409/) (2024)
- [DopUS-Net: Quality-Aware Robotic Ultrasound Imaging based on Doppler Signal](https://mlsanity.com/paper/2305.08938/) (2023)
- [Ultrasound Plane Pose Regression: Assessing Generalized Pose Coordinates in the Fetal Brain](https://mlsanity.com/paper/2301.08317/) (2023)
- [Autonomous Path Planning for Intercostal Robotic Ultrasound Imaging Using Reinforcement Learning](https://mlsanity.com/paper/2404.09927/) (2024)
- [UltraSeP: Sequence-aware Pre-training for Echocardiography Probe Movement Guidance](https://mlsanity.com/paper/2408.15026/) (2024)
- [End-to-End Framework Integrating Generative AI and Deep Reinforcement Learning for Autonomous Ultrasound Scanning](https://mlsanity.com/paper/2511.00114/) (2025)
- [SAMe: A Semantic Anatomy Mapping Engine for Robotic Ultrasound](https://mlsanity.com/paper/2604.25646/) (2026)
- [Scanning While Imagining: A Scene-Graph World Model for Robotic Ultrasound Navigation](https://mlsanity.com/paper/2609.32837/) (2026)
- [Ultrasound Operator Guidance Using World Modeling and Retrieval Based Action Planning](https://mlsanity.com/paper/2610.06008/) (2026)

## Abstract (arXiv)

Ultrasound (US) is one of the most common medical imaging modalities since it is radiation-free, low-cost, and real-time. In freehand US examinations, sonographers often navigate a US probe to visualize standard examination planes with rich diagnostic information. However, reproducibility and stability of the resulting images often suffer from intra- and inter-operator variation. Reinforcement learning (RL), as an interaction-based learning method, has demonstrated its effectiveness in visual navigating tasks; however, RL is limited in terms of generalization. To address this challenge, we propose a simulation-based RL framework for real-world navigation of US probes towards the standard longitudinal views of vessels. A UNet is used to provide binary masks from US images; thereby, the RL agent trained on simulated binary vessel images can be applied in real scenarios without further training. To accurately characterize actual states, a multi-modality state representation structure is introduced to facilitate the understanding of environments. Moreover, considering the characteristics of vessels, a novel standard view recognition approach based on the minimum bounding rectangle is proposed to terminate the searching process. To evaluate the effectiveness of the proposed method, the trained policy is validated virtually on 3D volumes of a volunteer's in-vivo carotid artery, and physically on custom-designed gel phantoms using robotic US. The results demonstrate that proposed approach can effectively and accurately navigate the probe towards the longitudinal view of vessels.