# PAg-NeRF: Towards fast and efficient end-to-end panoptic 3D representations for agricultural robotics arXiv: https://arxiv.org/abs/2309.05339 ยท HTML: https://mlsanity.com/paper/2309.05339/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Builds on 27 of its 38 references that are in the map. - First posted on arXiv 11 Sep 2023. ## Builds on (most cited first, 27 of 27) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022) - [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021) - [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022) - [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018) - [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021) - [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021) - [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020) - [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018) - [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019) - [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019) - [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021) - [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021) - [Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes](https://mlsanity.com/paper/2101.10994/) (2021) - [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021) - [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019) - [OneFormer: One Transformer to Rule Universal Image Segmentation](https://mlsanity.com/paper/2211.06220/) (2022) - [Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation](https://mlsanity.com/paper/2205.04334/) (2022) - [Panoptic Lifting for 3D Scene Understanding with Neural Fields](https://mlsanity.com/paper/2212.09802/) (2022) - [Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation](https://mlsanity.com/paper/2203.15224/) (2022) - [PermutoSDF: Fast Multi-View Reconstruction with Implicit Surfaces using Permutohedral Lattices](https://mlsanity.com/paper/2211.12562/) (2022) - [PATHoBot: A Robot for Glasshouse Crop Phenotyping and Intervention](https://mlsanity.com/paper/2010.16272/) (2020) - [Panoptic Mapping with Fruit Completion and Pose Estimation for Horticultural Robots](https://mlsanity.com/paper/2303.08923/) (2023) - [Detection of Single Grapevine Berries in Images Using Fully Convolutional Neural Networks](https://mlsanity.com/paper/1905.00458/) (2019) - [Towards Autonomous Visual Navigation in Arable Fields](https://mlsanity.com/paper/2109.11936/) (2021) - [Explicitly incorporating spatial information to recurrent networks for agriculture](https://mlsanity.com/paper/2206.13406/) (2022) ## Built on by (most cited first, 16 of 16) - [FruitNeRF: A Unified Neural Radiance Field based Fruit Counting Framework](https://mlsanity.com/paper/2408.06190/) (2024) - [AlignMiF: Geometry-Aligned Multimodal Implicit Field for LiDAR-Camera Joint Synthesis](https://mlsanity.com/paper/2402.17483/) (2024) - [Wheat3DGS: In-field 3D Reconstruction, Instance Segmentation and Phenotyping of Wheat Heads with Gaussian Splatting](https://mlsanity.com/paper/2504.06978/) (2025) - [PanicleNeRF: low-cost, high-precision in-field phenotypingof rice panicles with smartphone](https://mlsanity.com/paper/2408.02053/) (2024) - [A Survey on 3D Reconstruction Techniques in Plant Phenotyping: From Classical Methods to Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS), and Beyond](https://mlsanity.com/paper/2505.00737/) (2025) - [Multi-Spectral Gaussian Splatting with Neural Color Representation](https://mlsanity.com/paper/2506.03407/) (2025) - [Track Any Peppers: Weakly Supervised Sweet Pepper Tracking Using VLMs](https://mlsanity.com/paper/2411.06702/) (2024) - [FruitNeRF++: A Generalized Multi-Fruit Counting Method Utilizing Contrastive Learning and Neural Radiance Fields](https://mlsanity.com/paper/2505.19863/) (2025) - [CropCraft: Complete Structural Characterization of Crop Plants From Images](https://mlsanity.com/paper/2411.09693/) (2024) - [Demeter: A Parametric Model of Crop Plant Morphology from the Real World](https://mlsanity.com/paper/2510.16377/) (2025) - [Enhancing Single Image to 3D Generation using Gaussian Splatting and Hybrid Diffusion Priors](https://mlsanity.com/paper/2410.09467/) (2024) - [Botany-Bot: Digital Twin Monitoring of Occluded and Underleaf Plant Structures with Gaussian Splats](https://mlsanity.com/paper/2510.17783/) (2025) - [3D Hierarchical Panoptic Segmentation in Real Orchard Environments Across Different Sensors](https://mlsanity.com/paper/2503.13188/) (2025) - [DATR: Diffusion-based 3D Apple Tree Reconstruction Framework with Sparse-View](https://mlsanity.com/paper/2508.19508/) (2025) - [Still image and spatial-temporal tomato data enabling detection, segmentation, tracking, and video-instance segmentation using strong and weak labels](https://mlsanity.com/paper/2607.14934/) (2026) - [UAV3DCrop: Benchmarking 3D Reconstruction in Repeated Multi-Angle UAV Crop Surveys](https://mlsanity.com/paper/2608.06404/) (2026) ## Abstract (arXiv) Precise scene understanding is key for most robot monitoring and intervention tasks in agriculture. In this work we present PAg-NeRF which is a novel NeRF-based system that enables 3D panoptic scene understanding. Our representation is trained using an image sequence with noisy robot odometry poses and automatic panoptic predictions with inconsistent IDs between frames. Despite this noisy input, our system is able to output scene geometry, photo-realistic renders and 3D consistent panoptic representations with consistent instance IDs. We evaluate this novel system in a very challenging horticultural scenario and in doing so demonstrate an end-to-end trainable system that can make use of noisy robot poses rather than precise poses that have to be pre-calculated. Compared to a baseline approach the peak signal to noise ratio is improved from 21.34dB to 23.37dB while the panoptic quality improves from 56.65% to 70.08%. Furthermore, our approach is faster and can be tuned to improve inference time by more than a factor of 2 while being memory efficient with approximately 12 times fewer parameters.