# Horticultural Temporal Fruit Monitoring via 3D Instance Segmentation and Re-Identification using Colored Point Clouds

arXiv: https://arxiv.org/abs/2411.07799 · HTML: https://mlsanity.com/paper/2411.07799/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 59 references that are in the map.
- First posted on arXiv 12 Nov 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [OneFormer3D: One Transformer for Unified Point Cloud Segmentation](https://mlsanity.com/paper/2311.14405/) (2023)
- [LiDAR-based Panoptic Segmentation via Dynamic Shifting Network](https://mlsanity.com/paper/2011.11964/) (2020)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR-based Perception](https://mlsanity.com/paper/2109.05441/) (2021)
- [Advancements in Point Cloud Data Augmentation for Deep Learning: A Survey](https://mlsanity.com/paper/2308.12113/) (2023)
- [Spherical Mask: Coarse-to-Fine 3D Point Cloud Instance Segmentation with Spherical Representation](https://mlsanity.com/paper/2312.11269/) (2023)
- [PATHoBot: A Robot for Glasshouse Crop Phenotyping and Intervention](https://mlsanity.com/paper/2010.16272/) (2020)
- [Agricultural Robotics: The Future of Robotic Agriculture](https://mlsanity.com/paper/1806.06762/) (2018)
- [Position-Guided Point Cloud Panoptic Segmentation Transformer](https://mlsanity.com/paper/2303.13509/) (2023)
- [Scalable 3D Panoptic Segmentation As Superpoint Graph Clustering](https://mlsanity.com/paper/2401.06704/) (2024)
- [Fruit Detection, Segmentation and 3D Visualisation of Environments in Apple Orchards](https://mlsanity.com/paper/1911.12889/) (2019)
- [Behind the leaves -- Estimation of occluded grapevine berries with conditional generative adversarial networks](https://mlsanity.com/paper/2105.10325/) (2021)
- [A Review of Panoptic Segmentation for Mobile Mapping Point Clouds](https://mlsanity.com/paper/2304.13980/) (2023)
- [Semantic Segmentation of Fruits on Multi-sensor Fused Data in Natural Orchards](https://mlsanity.com/paper/2208.02483/) (2022)
- [A Ground Mobile Robot for Autonomous Terrestrial Laser Scanning-Based Field Phenotyping](https://mlsanity.com/paper/2404.04404/) (2024)

## Built on by (most cited first, 1 of 1)

- [GrowFields: Compositional 4D Neural Fields for Topology-Changing Plant Growth](https://mlsanity.com/paper/2607.03330/) (2026)

## Abstract (arXiv)

Accurate and consistent fruit monitoring over time is a key step toward automated agricultural production systems. However, this task is inherently difficult due to variations in fruit size, shape, occlusion, orientation, and the dynamic nature of orchards where fruits may appear or disappear between observations. In this article, we propose a novel method for fruit instance segmentation and re-identification on 3D terrestrial point clouds collected over time. Our approach directly operates on dense colored point clouds, capturing fine-grained 3D spatial detail. We segment individual fruits using a learning-based instance segmentation method applied directly to the point cloud. For each segmented fruit, we extract a compact and discriminative descriptor using a 3D sparse convolutional neural network. To track fruits across different times, we introduce an attention-based matching network that associates fruits with their counterparts from previous sessions. Matching is performed using a probabilistic assignment scheme, selecting the most likely associations across time. We evaluate our approach on real-world datasets of strawberries and apples, demonstrating that it outperforms existing methods in both instance segmentation and temporal re-identification, enabling robust and precise fruit monitoring across complex and dynamic orchard environments. Keywords = Agricultural Robotics, 3D Fruit Tracking, Instance Segmentation, Deep Learning , Point Clouds, Sparse Convolutional Networks, Temporal Monitoring