# Towards Scaling Marine Perception with Synthetic Data

arXiv: https://arxiv.org/abs/2609.20680 · HTML: https://mlsanity.com/paper/2609.20680/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 17 Sep 2026.

## Builds on (most cited first, 23 of 23)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information](https://mlsanity.com/paper/2402.13616/) (2024)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [Robot Operating System 2: Design, Architecture, and Uses In The Wild](https://mlsanity.com/paper/2211.07752/) (2022)
- [Driving in the Matrix: Can Virtual Worlds Replace Human-Generated Annotations for Real World Tasks?](https://mlsanity.com/paper/1610.01983/) (2016)
- [UNav-Sim: A Visually Realistic Underwater Robotics Simulator and Synthetic Data-generation Framework](https://mlsanity.com/paper/2310.11927/) (2023)
- [Stonefish: Supporting Machine Learning Research in Marine Robotics](https://mlsanity.com/paper/2502.11887/) (2025)
- [OceanSim: A GPU-Accelerated Underwater Robot Perception Simulation Framework](https://mlsanity.com/paper/2503.01074/) (2025)
- [Underwater Object Detection in the Era of Artificial Intelligence: Current, Challenge, and Future](https://mlsanity.com/paper/2410.05577/) (2024)
- [MarineGym: A High-Performance Reinforcement Learning Platform for Underwater Robotics](https://mlsanity.com/paper/2503.09203/) (2025)
- [Hyper3D: Efficient 3D Representation via Hybrid Triplane and Octree Feature for Enhanced 3D Shape Variational Auto-Encoders](https://mlsanity.com/paper/2503.10403/) (2025)
- [STARS: Zero-shot Sim-to-Real Transfer for Segmentation of Shipwrecks in Sonar Imagery](https://mlsanity.com/paper/2310.01667/) (2023)
- [Syn2Real Domain Generalization for Underwater Mine-like Object Detection Using Side-Scan Sonar](https://mlsanity.com/paper/2410.12953/) (2024)
- [Underwater Robotic Simulators Review for Autonomous System Development](https://mlsanity.com/paper/2504.06245/) (2025)
- [SurfSLAM: Sim-to-Real Underwater Stereo Reconstruction For Real-Time SLAM](https://mlsanity.com/paper/2601.10814/) (2026)
- [Infinite Leagues Under the Sea: Photorealistic 3D Underwater Terrain Generation by Latent Fractal Diffusion Models](https://mlsanity.com/paper/2503.06784/) (2025)
- [A Preview of HoloOcean 2.0](https://mlsanity.com/paper/2510.06160/) (2025)
- [SynTable: A Synthetic Data Generation Pipeline for Unseen Object Amodal Instance Segmentation of Cluttered Tabletop Scenes](https://mlsanity.com/paper/2307.07333/) (2023)
- [SonarSweep: Fusing Sonar and Vision for Robust 3D Reconstruction via Plane Sweeping](https://mlsanity.com/paper/2511.00392/) (2025)
- [LOTUSim: Multi-Domain Simulator for Marine Robotics](https://mlsanity.com/paper/2607.03072/) (2026)

## Abstract (arXiv)

Scalable machine learning in challenging underwater environments is strongly limited by the lack of labeled real-world training data. This data is often expensive and laborious to gather, making large-scale real-world data challenging to gather and curate. However, simulated data can help close the gap, enabling many learning-based tasks for underwater perception. In this work, we extend OceanSim, an IsaacSim-based underwater perception simulator, with a Synthetic Data Generation (SDG) pipeline for training models to be used in underwater scenarios. The proposed pipeline enables users to generate large, automatically labeled, photorealistic datasets with configurable scene appearance, structure, and sensor settings. We evaluate the pipeline on a real-world sea urchin detection task and study how different forms of synthetic scene variation affect sim-to-real performance. Based on these experiments, we discuss findings on our results, main limitations of the current pipeline and identify future directions for improving underwater rendering fidelity, scene diversity, and the evaluation of sim-to-real generalization. The open-source code can be found at https://github.com/umfieldrobotics/OceanSim.