# Semantic Segmentation of Fruits on Multi-sensor Fused Data in Natural Orchards

arXiv: https://arxiv.org/abs/2208.02483 · HTML: https://mlsanity.com/paper/2208.02483/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 52 references that are in the map.
- First posted on arXiv 4 Aug 2022.

## Builds on (most cited first, 18 of 18)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [Understanding Convolution for Semantic Segmentation](https://mlsanity.com/paper/1702.08502/) (2017)
- [Deep Semantic Segmentation of Natural and Medical Images: A Review](https://mlsanity.com/paper/1910.07655/) (2019)
- [EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation](https://mlsanity.com/paper/2106.15277/) (2021)
- [Pixel-level Extrinsic Self Calibration of High Resolution LiDAR and Camera in Targetless Environments](https://mlsanity.com/paper/2103.01627/) (2021)
- [RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/1906.00208/) (2019)
- [An Overview of Depth Cameras and Range Scanners Based on Time-of-Flight Technologies](https://mlsanity.com/paper/2012.06772/) (2020)
- [CamVox: A Low-cost and Accurate Lidar-assisted Visual SLAM System](https://mlsanity.com/paper/2011.11357/) (2020)
- [Fruit Detection, Segmentation and 3D Visualisation of Environments in Apple Orchards](https://mlsanity.com/paper/1911.12889/) (2019)
- [ACSC: Automatic Calibration for Non-repetitive Scanning Solid-State LiDAR and Camera Systems](https://mlsanity.com/paper/2011.08516/) (2020)
- [Visual Perception and Modelling in Unstructured Orchard for Apple Harvesting Robots](https://mlsanity.com/paper/1912.12555/) (2019)
- [Geometry-Aware Fruit Grasping Estimation for Robotic Harvesting in Orchards](https://mlsanity.com/paper/2112.04363/) (2021)
- [A procedure for automated tree pruning suggestion using LiDAR scans of fruit trees](https://mlsanity.com/paper/2102.03700/) (2021)

## Built on by (most cited first, 4 of 4)

- [High-fidelity 3D Reconstruction of Plants using Neural Radiance Field](https://mlsanity.com/paper/2311.04154/) (2023)
- [Exploring Accurate 3D Phenotyping in Greenhouse through Neural Radiance Fields](https://mlsanity.com/paper/2403.15981/) (2024)
- [Horticultural Temporal Fruit Monitoring via 3D Instance Segmentation and Re-Identification using Colored Point Clouds](https://mlsanity.com/paper/2411.07799/) (2024)
- [Multi-vision-based Picking Point Localisation of Target Fruit for Harvesting Robots](https://mlsanity.com/paper/2502.12406/) (2025)

## Abstract (arXiv)

Semantic segmentation is a fundamental task for agricultural robots to understand the surrounding environments in natural orchards. The recent development of the LiDAR techniques enables the robot to acquire accurate range measurements of the view in the unstructured orchards. Compared to RGB images, 3D point clouds have geometrical properties. By combining the LiDAR and camera, rich information on geometries and textures can be obtained. In this work, we propose a deep-learning-based segmentation method to perform accurate semantic segmentation on fused data from a LiDAR-Camera visual sensor. Two critical problems are explored and solved in this work. The first one is how to efficiently fused the texture and geometrical features from multi-sensor data. The second one is how to efficiently train the 3D segmentation network under severely imbalance class conditions. Moreover, an implementation of 3D segmentation in orchards including LiDAR-Camera data fusion, data collection and labelling, network training, and model inference is introduced in detail. In the experiment, we comprehensively analyze the network setup when dealing with highly unstructured and noisy point clouds acquired from an apple orchard. Overall, our proposed method achieves 86.2% mIoU on the segmentation of fruits on the high-resolution point cloud (100k-200k points). The experiment results show that the proposed method can perform accurate segmentation in real orchard environments.