# MACARONS: Mapping And Coverage Anticipation with RGB Online Self-Supervision

arXiv: https://arxiv.org/abs/2303.03315 · HTML: https://mlsanity.com/paper/2303.03315/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 89 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Builds on (most cited first, 30 of 57)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [GA-Net: Guided Aggregation Net for End-to-end Stereo Matching](https://mlsanity.com/paper/1904.06587/) (2019)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [Consistent Video Depth Estimation](https://mlsanity.com/paper/2004.15021/) (2020)
- [Learning to Recover 3D Scene Shape from a Single Image](https://mlsanity.com/paper/2012.09365/) (2020)
- [The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth](https://mlsanity.com/paper/2104.14540/) (2021)
- [Single-Image Depth Perception in the Wild](https://mlsanity.com/paper/1604.03901/) (2016)
- [Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Cameras](https://mlsanity.com/paper/1904.04998/) (2019)
- [Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos](https://mlsanity.com/paper/1811.06152/) (2018)
- [CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction](https://mlsanity.com/paper/1704.03489/) (2017)
- [CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM](https://mlsanity.com/paper/1804.00874/) (2018)
- [Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches](https://mlsanity.com/paper/1510.05970/) (2015)

## Built on by (most cited first, 19 of 19)

- [Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration](https://mlsanity.com/paper/2502.16652/) (2025)
- [NextBestPath: Efficient 3D Mapping of Unseen Environments](https://mlsanity.com/paper/2502.05378/) (2025)
- [SO-NeRF: Active View Planning for NeRF using Surrogate Objectives](https://mlsanity.com/paper/2312.03266/) (2023)
- [MAGICIAN: Efficient Long-Term Planning with Imagined Gaussians for Active Mapping](https://mlsanity.com/paper/2603.22650/) (2026)
- [WarpRF: Multi-View Consistency for Training-Free Uncertainty Quantification and Applications in Radiance Fields](https://mlsanity.com/paper/2506.22433/) (2025)
- [VIN-NBV: A View Introspection Network for Next-Best-View Selection](https://mlsanity.com/paper/2505.06219/) (2025)
- [SEA: Semantic Map Prediction for Active Exploration of Uncertain Areas](https://mlsanity.com/paper/2510.19766/) (2025)
- [SA-ResGS: Self-Augmented Residual 3D Gaussian Splatting for Next Best View Selection](https://mlsanity.com/paper/2601.03024/) (2026)
- [Hestia: Voxel-Face-Aware Hierarchical Next-Best-View Acquisition for Efficient 3D Reconstruction](https://mlsanity.com/paper/2508.01014/) (2025)
- [Active search and coverage using point-cloud reinforcement learning](https://mlsanity.com/paper/2312.11410/) (2023)
- [Paparazzo: Active Mapping of Moving 3D Objects](https://mlsanity.com/paper/2604.19556/) (2026)
- [Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration](https://mlsanity.com/paper/2605.22814/) (2026)
- [Deep Learning Approaches for 3D Medical Scene Completion: From Geometric Modeling to Generative Paradigms](https://mlsanity.com/paper/2606.24180/) (2026)
- [Iterative Hybrid Discrete-Continuous Viewpoint Planning for UAV Photogrammetry](https://mlsanity.com/paper/2608.05718/) (2026)
- [Diagnosing and Dynamically Filtering Occupancy World Models for Active Mapping](https://mlsanity.com/paper/2609.06820/) (2026)
- [Rethinking Learned Occupancy in Autonomous Active Mapping with Observation-Gated Filtering](https://mlsanity.com/paper/2609.09069/) (2026)
- [OpenFlyScan: A Quality-Guided Aerial Reconstruction System for Consumer Drones](https://mlsanity.com/paper/2609.24253/) (2026)
- [Multi-Scale Semantic Mapping in Urban Environments via Observation Calibration and Policy Dependence Regularization](https://mlsanity.com/paper/2609.34833/) (2026)
- [Matisse: Evidence-Space Reasoning for Active 3D Reconstruction](https://mlsanity.com/paper/2609.38746/) (2026)

## Abstract (arXiv)

We introduce a method that simultaneously learns to explore new large environments and to reconstruct them in 3D from color images only. This is closely related to the Next Best View problem (NBV), where one has to identify where to move the camera next to improve the coverage of an unknown scene. However, most of the current NBV methods rely on depth sensors, need 3D supervision and/or do not scale to large scenes. Our method requires only a color camera and no 3D supervision. It simultaneously learns in a self-supervised fashion to predict a "volume occupancy field" from color images and, from this field, to predict the NBV. Thanks to this approach, our method performs well on new scenes as it is not biased towards any training 3D data. We demonstrate this on a recent dataset made of various 3D scenes and show it performs even better than recent methods requiring a depth sensor, which is not a realistic assumption for outdoor scenes captured with a flying drone.