# SSL-NBV: A Self-Supervised-Learning-Based Next-Best-View algorithm for Efficient 3D Plant Reconstruction by a Robot

arXiv: https://arxiv.org/abs/2410.14790 · HTML: https://mlsanity.com/paper/2410.14790/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 49 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 12 of 12)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [A Deeper Look at Experience Replay](https://mlsanity.com/paper/1712.01275/) (2017)
- [BADGR: An Autonomous Self-Supervised Learning-Based Navigation System](https://mlsanity.com/paper/2002.05700/) (2020)
- [Self-supervised 6D Object Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/1909.10159/) (2019)
- [Learning 3D Shape Completion under Weak Supervision](https://mlsanity.com/paper/1805.07290/) (2018)
- [Supervised Learning of the Next-Best-View for 3D Object Reconstruction](https://mlsanity.com/paper/1905.05833/) (2019)
- [Viewpoint Planning for Fruit Size and Position Estimation](https://mlsanity.com/paper/2011.00275/) (2020)
- [3D Move to See: Multi-perspective visual servoing for improving object views with semantic segmentation](https://mlsanity.com/paper/1809.07896/) (2018)
- [Combining Local and Global Viewpoint Planning for Fruit Coverage](https://mlsanity.com/paper/2108.08114/) (2021)
- [Towards Active Robotic Vision in Agriculture: A Deep Learning Approach to Visual Servoing in Occluded and Unstructured Protected Cropping Environments](https://mlsanity.com/paper/1908.01885/) (2019)

## Built on by (most cited first, 1 of 1)

- [Review and Evaluation of Point-Cloud based Leaf Surface Reconstruction Methods for Agricultural Applications](https://mlsanity.com/paper/2604.03328/) (2026)

## Abstract (arXiv)

The 3D reconstruction of plants is challenging due to their complex shape causing many occlusions. Next-Best-View (NBV) methods address this by iteratively selecting new viewpoints to maximize information gain (IG). Deep-learning-based NBV (DL-NBV) methods demonstrate higher computational efficiency over classic voxel-based NBV approaches but current methods require extensive training using ground-truth plant models, making them impractical for real-world plants. These methods, moreover, rely on offline training with pre-collected data, limiting adaptability in changing agricultural environments. This paper proposes a self-supervised learning-based NBV method (SSL-NBV) that uses a deep neural network to predict the IG for candidate viewpoints. The method allows the robot to gather its own training data during task execution by comparing new 3D sensor data to the earlier gathered data and by employing weakly-supervised learning and experience replay for efficient online learning. Comprehensive evaluations were conducted in simulation and real-world environments using cross-validation. The results showed that SSL-NBV required fewer views for plant reconstruction than non-NBV methods and was over 800 times faster than a voxel-based method. SSL-NBV reduced training annotations by over 90% compared to a baseline DL-NBV. Furthermore, SSL-NBV could adapt to novel scenarios through online fine-tuning. Also using real plants, the results showed that the proposed method can learn to effectively plan new viewpoints for 3D plant reconstruction. Most importantly, SSL-NBV automated the entire network training and uses continuous online learning, allowing it to operate in changing agricultural environments.