# STARS: Zero-shot Sim-to-Real Transfer for Segmentation of Shipwrecks in Sonar Imagery

arXiv: https://arxiv.org/abs/2310.01667 · HTML: https://mlsanity.com/paper/2310.01667/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Builds on (most cited first, 20 of 20)

- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Towards Total Recall in Industrial Anomaly Detection](https://mlsanity.com/paper/2106.08265/) (2021)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/1906.00817/) (2019)
- [Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection](https://mlsanity.com/paper/1708.01642/) (2017)
- [PANDA: Adapting Pretrained Features for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2010.05903/) (2020)
- [HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation](https://mlsanity.com/paper/2204.13132/) (2022)
- [DeSTSeg: Segmentation Guided Denoising Student-Teacher for Anomaly Detection](https://mlsanity.com/paper/2211.11317/) (2022)
- [SePiCo: Semantic-Guided Pixel Contrast for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2204.08808/) (2022)
- [PODA: Prompt-driven Zero-shot Domain Adaptation](https://mlsanity.com/paper/2212.03241/) (2022)
- [Revisiting Image Pyramid Structure for High Resolution Salient Object Detection](https://mlsanity.com/paper/2209.09475/) (2022)
- [From Pixel to Patch: Synthesize Context-aware Features for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2009.12232/) (2020)
- [Deep Learning from Shallow Dives: Sonar Image Generation and Training for Underwater Object Detection](https://mlsanity.com/paper/1810.07990/) (2018)
- [Synthetic Sonar Image Simulation with Various Seabed Conditions for Automatic Target Recognition](https://mlsanity.com/paper/2210.10267/) (2022)

## Built on by (most cited first, 4 of 4)

- [Sonar-based Deep Learning in Underwater Robotics: Overview, Robustness and Challenges](https://mlsanity.com/paper/2412.11840/) (2024)
- [SonarSplat: Novel View Synthesis of Imaging Sonar via Gaussian Splatting](https://mlsanity.com/paper/2504.00159/) (2025)
- [ShipwreckFinder: A QGIS Tool for Shipwreck Detection in Multibeam Sonar Data](https://mlsanity.com/paper/2509.21386/) (2025)
- [Towards Scaling Marine Perception with Synthetic Data](https://mlsanity.com/paper/2609.20680/) (2026)

## Abstract (arXiv)

In this paper, we address the problem of sim-to-real transfer for object segmentation when there is no access to real examples of an object of interest during training, i.e. zero-shot sim-to-real transfer for segmentation. We focus on the application of shipwreck segmentation in side scan sonar imagery. Our novel segmentation network, STARS, addresses this challenge by fusing a predicted deformation field and anomaly volume, allowing it to generalize better to real sonar images and achieve more effective zero-shot sim-to-real transfer for image segmentation. We evaluate the sim-to-real transfer capabilities of our method on a real, expert-labeled side scan sonar dataset of shipwrecks collected from field work surveys with an autonomous underwater vehicle (AUV). STARS is trained entirely in simulation and performs zero-shot shipwreck segmentation with no additional fine-tuning on real data. Our method provides a significant 20% increase in segmentation performance for the targeted shipwreck class compared to the best baseline.