# Panoptic Mapping with Fruit Completion and Pose Estimation for Horticultural Robots

arXiv: https://arxiv.org/abs/2303.08923 · HTML: https://mlsanity.com/paper/2303.08923/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 36 references that are in the map.
- First posted on arXiv 15 Mar 2023.

## Builds on (most cited first, 13 of 13)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [PCN: Point Completion Network](https://mlsanity.com/paper/1808.00671/) (2018)
- [Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping](https://mlsanity.com/paper/1910.02490/) (2019)
- [Voxblox: Incremental 3D Euclidean Signed Distance Fields for On-Board MAV Planning](https://mlsanity.com/paper/1611.03631/) (2016)
- [FroDO: From Detections to 3D Objects](https://mlsanity.com/paper/2005.05125/) (2020)
- [Panoptic Multi-TSDFs: a Flexible Representation for Online Multi-resolution Volumetric Mapping and Long-term Dynamic Scene Consistency](https://mlsanity.com/paper/2109.10165/) (2021)
- [DSP-SLAM: Object Oriented SLAM with Deep Shape Priors](https://mlsanity.com/paper/2108.09481/) (2021)
- [PATHoBot: A Robot for Glasshouse Crop Phenotyping and Intervention](https://mlsanity.com/paper/2010.16272/) (2020)
- [Behind the leaves -- Estimation of occluded grapevine berries with conditional generative adversarial networks](https://mlsanity.com/paper/2105.10325/) (2021)
- [Fruit Mapping with Shape Completion for Autonomous Crop Monitoring](https://mlsanity.com/paper/2203.15489/) (2022)
- [Autonomous Sweet Pepper Harvesting for Protected Cropping Systems](https://mlsanity.com/paper/1706.02023/) (2017)
- [Peduncle Detection of Sweet Pepper for Autonomous Crop Harvesting - Combined Colour and 3D Information](https://mlsanity.com/paper/1701.08608/) (2017)

## Built on by (most cited first, 14 of 14)

- [PAg-NeRF: Towards fast and efficient end-to-end panoptic 3D representations for agricultural robotics](https://mlsanity.com/paper/2309.05339/) (2023)
- [Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environments](https://mlsanity.com/paper/2312.09138/) (2023)
- [Wheat3DGS: In-field 3D Reconstruction, Instance Segmentation and Phenotyping of Wheat Heads with Gaussian Splatting](https://mlsanity.com/paper/2504.06978/) (2025)
- [CF-PRNet: Coarse-to-Fine Prototype Refining Network for Point Cloud Completion and Reconstruction](https://mlsanity.com/paper/2409.08443/) (2024)
- [GrowSplat: Constructing Temporal Digital Twins of Plants with Gaussian Splats](https://mlsanity.com/paper/2505.10923/) (2025)
- [A Dataset and Benchmark for Shape Completion of Fruits for Agricultural Robotics](https://mlsanity.com/paper/2407.13304/) (2024)
- [High-throughput 3D shape completion of potato tubers on a harvester](https://mlsanity.com/paper/2407.21341/) (2024)
- [DM-OSVP++: One-Shot View Planning Using 3D Diffusion Models for Active RGB-Based Object Reconstruction](https://mlsanity.com/paper/2504.11674/) (2025)
- [Botany-Bot: Digital Twin Monitoring of Occluded and Underleaf Plant Structures with Gaussian Splats](https://mlsanity.com/paper/2510.17783/) (2025)
- [VAIR: Visuo-Acoustic Implicit Representations for Low-Cost, Multi-Modal Transparent Surface Reconstruction in Indoor Scenes](https://mlsanity.com/paper/2411.04963/) (2024)
- [3D Hierarchical Panoptic Segmentation in Real Orchard Environments Across Different Sensors](https://mlsanity.com/paper/2503.13188/) (2025)
- [PointRAFT: 3D deep learning for high-throughput prediction of potato tuber weight from partial point clouds](https://mlsanity.com/paper/2512.24193/) (2025)
- [SG-DOR: Learning Scene Graphs with Direction-Conditioned Occlusion Reasoning for Pepper Plants](https://mlsanity.com/paper/2603.06512/) (2026)
- [3D Reconstruction and Knowledge Distillation to Improve Multi-View Image Models to Explore Spike Volume Estimation in Wheat](https://mlsanity.com/paper/2605.20940/) (2026)

## Abstract (arXiv)

Monitoring plants and fruits at high resolution play a key role in the future of agriculture. Accurate 3D information can pave the way to a diverse number of robotic applications in agriculture ranging from autonomous harvesting to precise yield estimation. Obtaining such 3D information is non-trivial as agricultural environments are often repetitive and cluttered, and one has to account for the partial observability of fruit and plants. In this paper, we address the problem of jointly estimating complete 3D shapes of fruit and their pose in a 3D multi-resolution map built by a mobile robot. To this end, we propose an online multi-resolution panoptic mapping system where regions of interest are represented with a higher resolution. We exploit data to learn a general fruit shape representation that we use at inference time together with an occlusion-aware differentiable rendering pipeline to complete partial fruit observations and estimate the 7 DoF pose of each fruit in the map. The experiments presented in this paper evaluated both in the controlled environment and in a commercial greenhouse, show that our novel algorithm yields higher completion and pose estimation accuracy than existing methods, with an improvement of 41% in completion accuracy and 52% in pose estimation accuracy while keeping a low inference time of 0.6s in average. Codes are available at: https://github.com/PRBonn/HortiMapping.