# Ultrasound Operator Guidance Using World Modeling and Retrieval Based Action Planning

arXiv: https://arxiv.org/abs/2610.06008 · HTML: https://mlsanity.com/paper/2610.06008/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical and surgical world models, one of 11 directions in World models.
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Placed in

- World models / [Medical and surgical world models](https://mlsanity.com/wm/medical-and-surgical-world-models/)

## Builds on (most cited first, 22 of 22)

- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control](https://mlsanity.com/paper/1812.00568/) (2018)
- [Semi-parametric Topological Memory for Navigation](https://mlsanity.com/paper/1803.00653/) (2018)
- [Search on the Replay Buffer: Bridging Planning and Reinforcement Learning](https://mlsanity.com/paper/1906.05253/) (2019)
- [The Surprising Effectiveness of Representation Learning for Visual Imitation](https://mlsanity.com/paper/2112.01511/) (2021)
- [V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning](https://mlsanity.com/paper/2603.14482/) (2026)
- [ViNG: Learning Open-World Navigation with Visual Goals](https://mlsanity.com/paper/2012.09812/) (2020)
- [Automatic Probe Movement Guidance for Freehand Obstetric Ultrasound](https://mlsanity.com/paper/2007.04480/) (2020)
- [Cardiac Copilot: Automatic Probe Guidance for Echocardiography with World Model](https://mlsanity.com/paper/2406.13165/) (2024)
- [EchoWorld: Learning Motion-Aware World Models for Echocardiography Probe Guidance](https://mlsanity.com/paper/2504.13065/) (2025)
- [Offline Imitation Learning Through Graph Search and Retrieval](https://mlsanity.com/paper/2407.15403/) (2024)
- [EchoJEPA: A Latent Predictive Foundation Model for Echocardiography](https://mlsanity.com/paper/2602.02603/) (2026)
- [VesNet-RL: Simulation-based Reinforcement Learning for Real-World US Probe Navigation](https://mlsanity.com/paper/2205.06676/) (2022)
- [Model-based trajectory stitching for improved behavioural cloning and its applications](https://mlsanity.com/paper/2212.04280/) (2022)
- [RT-Cache: Training-Free Retrieval for Real-Time Manipulation](https://mlsanity.com/paper/2505.09040/) (2025)
- [PALMER: Perception-Action Loop with Memory for Long-Horizon Planning](https://mlsanity.com/paper/2212.04581/) (2022)
- [UltraSeP: Sequence-aware Pre-training for Echocardiography Probe Movement Guidance](https://mlsanity.com/paper/2408.15026/) (2024)
- [UltraHiT: A Hierarchical Transformer Architecture for Generalizable Internal Carotid Artery Robotic Ultrasonography](https://mlsanity.com/paper/2509.13832/) (2025)
- [Action-Conditioned World Model for Goal Plane Probe Guidance in Robotic Ultrasound](https://mlsanity.com/paper/2607.21918/) (2026)
- [RAG-RUSS: A Retrieval-Augmented Robotic Ultrasound for Autonomous Carotid Examination](https://mlsanity.com/paper/2603.01153/) (2026)

## Abstract (arXiv)

Ultrasound is widely used, but acquisition quality is heavily dependent on the operator's knowledge and expertise. With demand for examinations outpacing the supply of trained sonographers, operator-guidance systems aim to close this gap by instructing a less trained user how to move the probe toward a target view. In this paper, we propose a retrieval-induced latent transition model for ultrasound acquisition dynamics, formulating ultrasound operator guidance as multi-step planning and retrieval in a world model. Using a V-JEPA 2.1 backbone, observations are first encoded into a latent space where anatomically related views lie close together. We then retrieve similar views from a reference database containing encoded latent states and corresponding probe positions and orientations. Rather than learning a parametric transition function, we directly use physically executed transitions from the database to establish our nonparametric, retrieval-induced transition model that supports receding-horizon planning. At deployment, guidance is generated from the live ultrasound image feed alone, without any probe tracking hardware. Applied to carotid ultrasound, the proposed planner reaches the target view in 86% of retrospective closed-loop episodes, versus 52% and 43% for representative baselines, outperforming both on every target view, including the challenging longitudinal internal and external carotid artery views. A prospective feasibility study on unseen volunteers, run in real time on a CPU using distillation, reaches 83% target-view reachability. Because planning is driven by proximity to any encodable goal latent, the same world model can navigate back to any previously acquired, patient-specific frame, supporting reproducible longitudinal imaging for e.g. perioperative or follow-up monitoring.