# 3D Hierarchical Panoptic Segmentation in Real Orchard Environments Across Different Sensors

arXiv: https://arxiv.org/abs/2503.13188 · HTML: https://mlsanity.com/paper/2503.13188/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 49 references that are in the map.
- First posted on arXiv 17 Mar 2025.

## Builds on (most cited first, 20 of 20)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentation](https://mlsanity.com/paper/2103.14962/) (2021)
- [Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth](https://mlsanity.com/paper/1906.11109/) (2019)
- [EfficientLPS: Efficient LiDAR Panoptic Segmentation](https://mlsanity.com/paper/2102.08009/) (2021)
- [Real-time Semantic Segmentation of Crop and Weed for Precision Agriculture Robots Leveraging Background Knowledge in CNNs](https://mlsanity.com/paper/1709.06764/) (2017)
- [Panoster: End-to-end Panoptic Segmentation of LiDAR Point Clouds](https://mlsanity.com/paper/2010.15157/) (2020)
- [MinneApple: A Benchmark Dataset for Apple Detection and Segmentation](https://mlsanity.com/paper/1909.06441/) (2019)
- [PhenoBench -- A Large Dataset and Benchmarks for Semantic Image Interpretation in the Agricultural Domain](https://mlsanity.com/paper/2306.04557/) (2023)
- [PATHoBot: A Robot for Glasshouse Crop Phenotyping and Intervention](https://mlsanity.com/paper/2010.16272/) (2020)
- [Hierarchical Approach for Joint Semantic, Plant Instance, and Leaf Instance Segmentation in the Agricultural Domain](https://mlsanity.com/paper/2210.07879/) (2022)
- [PAg-NeRF: Towards fast and efficient end-to-end panoptic 3D representations for agricultural robotics](https://mlsanity.com/paper/2309.05339/) (2023)
- [GrowliFlower: An image time series dataset for GROWth analysis of cauLIFLOWER](https://mlsanity.com/paper/2204.00294/) (2022)
- [Panoptic Mapping with Fruit Completion and Pose Estimation for Horticultural Robots](https://mlsanity.com/paper/2303.08923/) (2023)
- [Joint Stem Detection and Crop-Weed Classification for Plant-specific Treatment in Precision Farming](https://mlsanity.com/paper/1806.03413/) (2018)
- [BonnBeetClouds3D: A Dataset Towards Point Cloud-based Organ-level Phenotyping of Sugar Beet Plants under Field Conditions](https://mlsanity.com/paper/2312.14706/) (2023)
- [Zoom in on the Plant: Fine-grained Analysis of Leaf, Stem and Vein Instances](https://mlsanity.com/paper/2312.08805/) (2023)
- [High-throughput 3D shape completion of potato tubers on a harvester](https://mlsanity.com/paper/2407.21341/) (2024)

## Abstract (arXiv)

Crop yield estimation is a relevant problem in agriculture, because an accurate yield estimate can support farmers' decisions on harvesting or precision intervention. Robots can help to automate this process. To do so, they need to be able to perceive the surrounding environment to identify target objects such as trees and plants. In this paper, we introduce a novel approach to address the problem of hierarchical panoptic segmentation of apple orchards on 3D data from different sensors. Our approach is able to simultaneously provide semantic segmentation, instance segmentation of trunks and fruits, and instance segmentation of trees (a trunk with its fruits). This allows us to identify relevant information such as individual plants, fruits, and trunks, and capture the relationship among them, such as precisely estimate the number of fruits associated to each tree in an orchard. To efficiently evaluate our approach for hierarchical panoptic segmentation, we provide a dataset designed specifically for this task. Our dataset is recorded in Bonn, Germany, in a real apple orchard with a variety of sensors, spanning from a terrestrial laser scanner to a RGB-D camera mounted on different robots platforms. The experiments show that our approach surpasses state-of-the-art approaches in 3D panoptic segmentation in the agricultural domain, while also providing full hierarchical panoptic segmentation. Our dataset is publicly available at https://www.ipb.uni-bonn.de/data/hops/. The open-source implementation of our approach is available at https://github.com/PRBonn/hapt3D.