# Scanning While Imagining: A Scene-Graph World Model for Robotic Ultrasound Navigation

arXiv: https://arxiv.org/abs/2609.32837 · HTML: https://mlsanity.com/paper/2609.32837/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical and surgical world models, one of 11 directions in World models.
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Placed in

- World models / [Medical and surgical world models](https://mlsanity.com/wm/medical-and-surgical-world-models/)

## Builds on (most cited first, 26 of 26)

- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [Navigation World Models](https://mlsanity.com/paper/2412.03572/) (2024)
- [Contrastive Learning of Structured World Models](https://mlsanity.com/paper/1911.12247/) (2019)
- [3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans](https://mlsanity.com/paper/2002.06289/) (2020)
- [Robotic Ultrasound Imaging: State-of-the-Art and Future Perspectives](https://mlsanity.com/paper/2307.05545/) (2023)
- [Autonomous Robotic Screening of Tubular Structures based only on Real-Time Ultrasound Imaging Feedback](https://mlsanity.com/paper/2011.00099/) (2020)
- [Autonomous Navigation of an Ultrasound Probe Towards Standard Scan Planes with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.00718/) (2021)
- [Ultrasound-Guided Robotic Navigation with Deep Reinforcement Learning](https://mlsanity.com/paper/2003.13321/) (2020)
- [Machine Learning in Robotic Ultrasound Imaging: Challenges and Perspectives](https://mlsanity.com/paper/2401.02376/) (2024)
- [EchoWorld: Learning Motion-Aware World Models for Echocardiography Probe Guidance](https://mlsanity.com/paper/2504.13065/) (2025)
- [Intelligent Robotic Sonographer: Mutual Information-based Disentangled Reward Learning from Few Demonstrations](https://mlsanity.com/paper/2307.03705/) (2023)
- [VesNet-RL: Simulation-based Reinforcement Learning for Real-World US Probe Navigation](https://mlsanity.com/paper/2205.06676/) (2022)
- [Skeleton Graph-based Ultrasound-CT Non-rigid Registration](https://mlsanity.com/paper/2305.08228/) (2023)
- [Speckle2Self: Self-Supervised Ultrasound Speckle Reduction Without Clean Data](https://mlsanity.com/paper/2507.06828/) (2025)
- [CADS: A Comprehensive Anatomical Dataset and Segmentation for Whole-Body Anatomy in Computed Tomography](https://mlsanity.com/paper/2507.22953/) (2025)
- [Robotic Sonographer: Autonomous Robotic Ultrasound using Domain Expertise in Bayesian Optimization](https://mlsanity.com/paper/2307.02442/) (2023)
- [Class-Aware Cartilage Segmentation for Autonomous US-CT Registration in Robotic Intercostal Ultrasound Imaging](https://mlsanity.com/paper/2406.04100/) (2024)
- [DreamReg: Belief-Driven World Model for 2D-3D Ultrasound Registration](https://mlsanity.com/paper/2606.18825/) (2026)
- [Needle Segmentation Using GAN: Restoring Thin Instrument Visibility in Robotic Ultrasound](https://mlsanity.com/paper/2407.18357/) (2024)
- [SonoGym: High Performance Simulation for Challenging Surgical Tasks with Robotic Ultrasound](https://mlsanity.com/paper/2507.01152/) (2025)
- [Action-Conditioned World Model for Goal Plane Probe Guidance in Robotic Ultrasound](https://mlsanity.com/paper/2607.21918/) (2026)

## Abstract (arXiv)

Ultrasound (US) acquisition depends on the operator's ability to interpret anatomy and anticipate how the view will change with probe motion. Many robotic US navigation methods select actions without explicitly predicting these anatomical changes. We propose SonoGraph-WM, an action- and goal-conditioned world model for anticipatory probe navigation. The model represents anatomy as scene graphs (SGs), capturing visible structures, their geometry, and spatial relationships without synthesizing US images. Given a history of SGs and probe poses, a unified Transformer jointly predicts future SGs and poses. A receding-horizon planner recursively imagines candidate trajectories, selects the shortest predicted path reaching a goal graph, and follows it over a short execution horizon before replanning from new observations. To reduce reliance on tracked and anatomically annotated US sequences, we generate aligned SG--pose training data from computed tomography (CT) label maps along surface-constrained probe trajectories. On four held-out CT cases, spatial relation F1 remains above 93% over 20 prediction steps, and closed-loop navigation achieves 77.50% and 75.00% success for the gallbladder and pancreas, respectively, using annotation-derived SGs. In robot--phantom navigation experiments with label-map-derived SGs, the planner reached the target view in 73.7% of trials. These findings support CT-supervised anatomical world modeling for probe planning and highlight the importance of frequent observation updates for reliable navigation. Project Page: https://noseefood.github.io/us-sonograph-wm/