# EchoPT: A Pretrained Transformer Architecture that Predicts 2D In-Air Sonar Images for Mobile Robotics

arXiv: https://arxiv.org/abs/2405.12573 · HTML: https://mlsanity.com/paper/2405.12573/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical and surgical world models, one of 11 directions in World models.
- Builds on 12 of its 45 references that are in the map.
- First posted on arXiv 21 May 2024.

## Placed in

- World models / [Medical and surgical world models](https://mlsanity.com/wm/medical-and-surgical-world-models/)

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Spherical Transformer for LiDAR-based 3D Recognition](https://mlsanity.com/paper/2303.12766/) (2023)
- [Self-supervised Video Transformer](https://mlsanity.com/paper/2112.01514/) (2021)
- [Effective Rotation-invariant Point CNN with Spherical Harmonics kernels](https://mlsanity.com/paper/1906.11555/) (2019)
- [VisualEchoes: Spatial Image Representation Learning through Echolocation](https://mlsanity.com/paper/2005.01616/) (2020)
- [BatVision: Learning to See 3D Spatial Layout with Two Ears](https://mlsanity.com/paper/1912.07011/) (2019)
- [Beyond Image to Depth: Improving Depth Prediction using Echoes](https://mlsanity.com/paper/2103.08468/) (2021)
- [A Convolutional Vision Transformer for Semantic Segmentation of Side-Scan Sonar Data](https://mlsanity.com/paper/2302.12416/) (2023)
- [Adaptive Acoustic Flow-Based Navigation with 3D Sonar Sensor Fusion](https://mlsanity.com/paper/2208.10823/) (2022)
- [Real-Time Sonar Fusion for Layered Navigation Controller](https://mlsanity.com/paper/2208.10834/) (2022)

## Built on by (most cited first, 1 of 1)

- [Predictive Modeling of Maritime Radar Data Using Transformer Architecture](https://mlsanity.com/paper/2512.17098/) (2025)

## Abstract (arXiv)

The predictive brain hypothesis suggests that perception can be interpreted as the process of minimizing the error between predicted perception tokens generated by an internal world model and actual sensory input tokens. When implementing working examples of this hypothesis in the context of in-air sonar, significant difficulties arise due to the sparse nature of the reflection model that governs ultrasonic sensing. Despite these challenges, creating consistent world models using sonar data is crucial for implementing predictive processing of ultrasound data in robotics. In an effort to enable robust robot behavior using ultrasound as the sole exteroceptive sensor modality, this paper introduces EchoPT, a pretrained transformer architecture designed to predict 2D sonar images from previous sensory data and robot ego-motion information. We detail the transformer architecture that drives EchoPT and compare the performance of our model to several state-of-the-art techniques. In addition to presenting and evaluating our EchoPT model, we demonstrate the effectiveness of this predictive perception approach in two robotic tasks.