mlsanityData to 7 Oct 2026

MACARONS: Mapping And Coverage Anticipation with RGB Online Self-Supervision

arXiv 2303.03315 ↗First posted 6 Mar 2023cs.CV cs.AI cs.ROcited by 18

Not on the map. Why?

Abstract (arXiv)

We introduce a method that simultaneously learns to explore new large environments and to reconstruct them in 3D from color images only. This is closely related to the Next Best View problem (NBV), where one has to identify where to move the camera next to improve the coverage of an unknown scene. However, most of the current NBV methods rely on depth sensors, need 3D supervision and/or do not scale to large scenes. Our method requires only a color camera and no 3D supervision. It simultaneously learns in a self-supervised fashion to predict a "volume occupancy field" from color images and, from this field, to predict the NBV. Thanks to this approach, our method performs well on new scenes as it is not biased towards any training 3D data. We demonstrate this on a recent dataset made of various 3D scenes and show it performs even better than recent methods requiring a depth sensor, which is not a realistic assumption for outdoor scenes captured with a flying drone.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 57 of its 89 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

57 of 89 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  8. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  9. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  10. 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 819
  11. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  12. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
38 more
  1. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 721
  2. 2018Pyramid Stereo Matching Networkcited by 486
  3. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  4. 2016Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescuecited by 368
  5. 2017End-to-End Learning of Geometry and Context for Deep Stereo Regressioncited by 359
  6. 2018GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Posecited by 341
  7. 2019Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Videocited by 244
  8. 2019GA-Net: Guided Aggregation Net for End-to-end Stereo Matchingcited by 242
  9. 2018Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentationcited by 222
  10. 2020Consistent Video Depth Estimationcited by 220
  11. 2020Learning to Recover 3D Scene Shape from a Single Imagecited by 214
  12. 2021The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depthcited by 190
  13. 2016Single-Image Depth Perception in the Wildcited by 179
  14. 2019Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Camerascited by 170
  15. 2018Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videoscited by 155
  16. 2017CNN-SLAM: Real-time dense monocular SLAM with learned depth predictioncited by 152
  17. 2015Stereo Matching by Training a Convolutional Neural Network to Compare Image Patchescited by 151
  18. 2018CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAMcited by 151
  19. 2020Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidancecited by 148
  20. 2020Feature-metric Loss for Self-supervised Learning of Depth and Egomotioncited by 140
  21. 2022MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformercited by 136
  22. 2018Learning Depth with Convolutional Spatial Propagation Networkcited by 135
  23. 2019Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Cameracited by 130
  24. 2019Learning the Depths of Moving People by Watching Frozen Peoplecited by 120
  25. 2019Domain-invariant Stereo Matching Networkscited by 108
  26. 2021Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Mergingcited by 104
  27. 2020Unsupervised Monocular Depth Learning in Dynamic Scenescited by 84
  28. 2016Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networkscited by 72
  29. 2020Don't Forget The Past: Recurrent Depth Estimation from Monocular Videocited by 71
  30. 2019Neural RGB->D Sensing: Depth and Uncertainty from a Video Cameracited by 69
  31. 2022Multi-Frame Self-Supervised Depth with Transformerscited by 64
  32. 2019Exploiting temporal consistency for real-time video depth estimationcited by 61
  33. 2020MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Cameracited by 59
  34. 2019Recurrent Neural Network for (Un-)supervised Learning of Monocular VideoVisual Odometry and Depthcited by 50
  35. 2019Multi-View Stereo by Temporal Nonparametric Fusioncited by 46
  36. 2020Distilled Semantics for Comprehensive Scene Understanding from Videoscited by 39
  37. 2019Supervised Learning of the Next-Best-View for 3D Object Reconstructioncited by 23
  38. 2019Video Depth Estimation by Fusing Flow-to-Depth Proposalscited by 23

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 0; 2025: 5; 2026: 11.

18 papers.

Sort:

By area:

  1. 2025Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registrationcited by 50
  2. 2025NextBestPath: Efficient 3D Mapping of Unseen Environmentscited by 13
  3. 2023SO-NeRF: Active View Planning for NeRF using Surrogate Objectivescited by 5
  4. 2026MAGICIAN: Efficient Long-Term Planning with Imagined Gaussians for Active Mappingcited by 5
  5. 2025VIN-NBV: A View Introspection Network for Next-Best-View Selectioncited by 2
  6. 2025WarpRF: Multi-View Consistency for Training-Free Uncertainty Quantification and Applications in Radiance Fieldscited by 2
  7. 2026SA-ResGS: Self-Augmented Residual 3D Gaussian Splatting for Next Best View Selectioncited by 2
  8. 2025Hestia: Voxel-Face-Aware Hierarchical Next-Best-View Acquisition for Efficient 3D Reconstructioncited by 1
  9. 2023Active search and coverage using point-cloud reinforcement learningcited by 0
  10. 2026Paparazzo: Active Mapping of Moving 3D Objectscited by 0
  11. 2026Remember to be Curious: Episodic Context and Persistent Worlds for 3D Explorationcited by 0
  12. 2026Deep Learning Approaches for 3D Medical Scene Completion: From Geometric Modeling to Generative Paradigmscited by 0
6 more
  1. 2026Iterative Hybrid Discrete-Continuous Viewpoint Planning for UAV Photogrammetrycited by 0
  2. 2026Diagnosing and Dynamically Filtering Occupancy World Models for Active Mappingcited by 0
  3. 2026Rethinking Learned Occupancy in Autonomous Active Mapping with Observation-Gated Filteringcited by 0
  4. 2026OpenFlyScan: A Quality-Guided Aerial Reconstruction System for Consumer Dronescited by 0
  5. 2026Multi-Scale Semantic Mapping in Urban Environments via Observation Calibration and Policy Dependence Regularizationcited by 0
  6. 2026Matisse: Evidence-Space Reasoning for Active 3D Reconstructioncited by 0