# MARVIS: Motion & Geometry Aware Real and Virtual Image Segmentation

arXiv: https://arxiv.org/abs/2403.09850 · HTML: https://mlsanity.com/paper/2403.09850/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [UNeXt: MLP-based Rapid Medical Image Segmentation Network](https://mlsanity.com/paper/2203.04967/) (2022)
- [Pyramid Attention Network for Semantic Segmentation](https://mlsanity.com/paper/1805.10180/) (2018)
- [GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning](https://mlsanity.com/paper/1810.05762/) (2018)
- [UDepth: Fast Monocular Depth Estimation for Visually-guided Underwater Robots](https://mlsanity.com/paper/2209.12358/) (2022)
- [UNav-Sim: A Visually Realistic Underwater Robotics Simulator and Synthetic Data-generation Framework](https://mlsanity.com/paper/2310.11927/) (2023)
- [DAVE Aquatic Virtual Environment: Toward a General Underwater Robotics Simulator](https://mlsanity.com/paper/2209.02862/) (2022)
- [OysterNet: Enhanced Oyster Detection Using Simulation](https://mlsanity.com/paper/2209.08176/) (2022)
- [OysterSim: Underwater Simulation for Enhancing Oyster Reef Monitoring](https://mlsanity.com/paper/2209.09395/) (2022)
- [Detecting and Counting Oysters](https://mlsanity.com/paper/2105.09758/) (2021)
- [ChatSim: Underwater Simulation with Natural Language Prompting](https://mlsanity.com/paper/2308.04029/) (2023)
- [SeaDroneSim: Simulation of Aerial Images for Detection of Objects Above Water](https://mlsanity.com/paper/2210.16107/) (2022)
- [Whale Detection Enhancement through Synthetic Satellite Images](https://mlsanity.com/paper/2308.07766/) (2023)

## Built on by (most cited first, 3 of 3)

- [Event3DGS: Event-Based 3D Gaussian Splatting for High-Speed Robot Egomotion](https://mlsanity.com/paper/2406.02972/) (2024)
- [ODYSSEE: Oyster Detection Yielded by Sensor Systems on Edge Electronics](https://mlsanity.com/paper/2409.07003/) (2024)
- [WaterGen: Decoupling Scene and Medium in Underwater Image Generation](https://mlsanity.com/paper/2606.31147/) (2026)

## Abstract (arXiv)

Tasks such as autonomous navigation, 3D reconstruction, and object recognition near the water surfaces are crucial in marine robotics applications. However, challenges arise due to dynamic disturbances, e.g., light reflections and refraction from the random air-water interface, irregular liquid flow, and similar factors, which can lead to potential failures in perception and navigation systems. Traditional computer vision algorithms struggle to differentiate between real and virtual image regions, significantly complicating tasks. A virtual image region is an apparent representation formed by the redirection of light rays, typically through reflection or refraction, creating the illusion of an object's presence without its actual physical location. This work proposes a novel approach for segmentation on real and virtual image regions, exploiting synthetic images combined with domain-invariant information, a Motion Entropy Kernel, and Epipolar Geometric Consistency. Our segmentation network does not need to be re-trained if the domain changes. We show this by deploying the same segmentation network in two different domains: simulation and the real world. By creating realistic synthetic images that mimic the complexities of the water surface, we provide fine-grained training data for our network (MARVIS) to discern between real and virtual images effectively. By motion & geometry-aware design choices and through comprehensive experimental analysis, we achieve state-of-the-art real-virtual image segmentation performance in unseen real world domain, achieving an IoU over 78% and a F1-Score over 86% while ensuring a small computational footprint. MARVIS offers over 43 FPS (8 FPS) inference rates on a single GPU (CPU core). Our code and dataset are available here https://github.com/jiayi-wu-umd/MARVIS.