# SeePerSea: Multi-modal Perception Dataset of In-water Objects for Autonomous Surface Vehicles

arXiv: https://arxiv.org/abs/2404.18411 · HTML: https://mlsanity.com/paper/2404.18411/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 62 references that are in the map.
- First posted on arXiv 29 Apr 2024.

## Builds on (most cited first, 27 of 27)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information](https://mlsanity.com/paper/2402.13616/) (2024)
- [Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection](https://mlsanity.com/paper/2012.15712/) (2020)
- [CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection](https://mlsanity.com/paper/2009.00784/) (2020)
- [PENet: Towards Precise and Efficient Image Guided Depth Completion](https://mlsanity.com/paper/2103.00783/) (2021)
- [Focal Sparse Convolutional Networks for 3D Object Detection](https://mlsanity.com/paper/2204.12463/) (2022)
- [Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving](https://mlsanity.com/paper/2210.08061/) (2022)
- [Transformation-Equivariant 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/2211.11962/) (2022)
- [Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection](https://mlsanity.com/paper/2205.14951/) (2022)
- [3D BAT: A Semi-Automatic, Web-based 3D Annotation Toolbox for Full-Surround, Multi-Modal Data Streams](https://mlsanity.com/paper/1905.00525/) (2019)
- [MODS -- A USV-oriented object detection and obstacle segmentation benchmark](https://mlsanity.com/paper/2105.02359/) (2021)
- [Automatic Extrinsic Calibration Method for LiDAR and Camera Sensor Setups](https://mlsanity.com/paper/2101.04431/) (2021)
- [Fast image-based obstacle detection from unmanned surface vehicles](https://mlsanity.com/paper/1503.01918/) (2015)
- [Stereo obstacle detection for unmanned surface vehicles by IMU-assisted semantic segmentation](https://mlsanity.com/paper/1802.07956/) (2018)
- [Are We Ready for Unmanned Surface Vehicles in Inland Waterways? The USVInland Multisensor Dataset and Benchmark](https://mlsanity.com/paper/2103.05383/) (2021)
- [MassMIND: Massachusetts Maritime INfrared Dataset](https://mlsanity.com/paper/2209.04097/) (2022)
- [Pohang Canal Dataset: A Multimodal Maritime Dataset for Autonomous Navigation in Restricted Waters](https://mlsanity.com/paper/2303.05555/) (2023)

## Built on by (most cited first, 2 of 2)

- [FlowMo-WM: A World Model with Object Momentum and Hidden Ambient Drift](https://mlsanity.com/paper/2606.13817/) (2026)
- [MMUSV-Sim: A Perception-Oriented Simulation and Data-Generation Platform for Multi-USV Cooperative Perception](https://mlsanity.com/paper/2608.14207/) (2026)

## Abstract (arXiv)

This paper introduces the first publicly accessible labeled multi-modal perception dataset for autonomous maritime navigation, focusing on in-water obstacles within the aquatic environment to enhance situational awareness for Autonomous Surface Vehicles (ASVs). This dataset, collected over 4 years and consisting of diverse objects encountered under varying environmental conditions, aims to bridge the research gap in ASVs by providing a multi-modal, annotated, and ego-centric perception dataset, for object detection and classification. We also show the applicability of the proposed dataset by training and testing current deep learning-based open-source perception algorithms that have shown success in the autonomous ground vehicle domain. With the training and testing results, we discuss open challenges for existing datasets and methods, identifying future research directions. We expect that our dataset will contribute to the development of future marine autonomy pipelines and marine (field) robotics. This dataset is open source and found at https://seepersea.github.io/.