# Comparing YOLOv8 and Mask R-CNN for instance segmentation in complex orchard environments

arXiv: https://arxiv.org/abs/2312.07935 · HTML: https://mlsanity.com/paper/2312.07935/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 111 references that are in the map.
- First posted on arXiv 13 Dec 2023.

## Builds on (most cited first, 9 of 9)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [U-Net and its variants for medical image segmentation: theory and applications](https://mlsanity.com/paper/2011.01118/) (2020)
- [A Survey on Instance Segmentation: State of the art](https://mlsanity.com/paper/2007.00047/) (2020)
- [Optimizing the Trade-off between Single-Stage and Two-Stage Object Detectors using Image Difficulty Prediction](https://mlsanity.com/paper/1803.08707/) (2018)
- [Immature Green Apple Detection and Sizing in Commercial Orchards using YOLOv8 and Shape Fitting Techniques](https://mlsanity.com/paper/2401.08629/) (2023)
- [Field Evaluations of A Deep Learning-based Intelligent Spraying Robot with Flow Control for Pear Orchards](https://mlsanity.com/paper/2102.07313/) (2021)
- [Toward Robotic Weed Control: Detection of Nutsedge Weed in Bermudagrass Turf Using Inaccurate and Insufficient Training Data](https://mlsanity.com/paper/2106.08897/) (2021)
- [Volume and leaf area calculation of cabbage with a neural network-based instance segmentation](https://mlsanity.com/paper/2104.05284/) (2021)

## Built on by (most cited first, 15 of 15)

- [YOLO26: Key Architectural Enhancements and Performance Benchmarking for Real-Time Object Detection](https://mlsanity.com/paper/2509.25164/) (2025)
- [Comparing YOLOv11 and YOLOv8 for instance segmentation of occluded and non-occluded immature green fruits in complex orchard environment](https://mlsanity.com/paper/2410.19869/) (2024)
- [Zero-Shot Automatic Annotation and Instance Segmentation using LLM-Generated Datasets: Eliminating Field Imaging and Manual Annotation for Deep Learning Model Development](https://mlsanity.com/paper/2411.11285/) (2024)
- [Improved YOLOv12 with LLM-Generated Synthetic Data for Enhanced Apple Detection and Benchmarking Against YOLOv11 and YOLOv10](https://mlsanity.com/paper/2503.00057/) (2025)
- [Joint 3D Point Cloud Segmentation using Real-Sim Loop: From Panels to Trees and Branches](https://mlsanity.com/paper/2503.05630/) (2025)
- [Improved Object-Based Style Transfer with Single Deep Network](https://mlsanity.com/paper/2404.09461/) (2024)
- [A Computer Vision Approach for Autonomous Cars to Drive Safe at Construction Zone](https://mlsanity.com/paper/2409.15809/) (2024)
- [A versatile machine learning workflow for high-throughput analysis of supported metal catalyst particles](https://mlsanity.com/paper/2410.01213/) (2024)
- [Review of Fruit Tree Image Segmentation](https://mlsanity.com/paper/2412.14631/) (2024)
- [Falcon: Fractional Alternating Cut with Overcoming Minima in Unsupervised Segmentation](https://mlsanity.com/paper/2504.05613/) (2025)
- [From Semantic To Instance: A Semi-Self-Supervised Learning Approach](https://mlsanity.com/paper/2506.16563/) (2025)
- [Deep Learning Perspective of Scene Understanding in Autonomous Robots](https://mlsanity.com/paper/2512.14020/) (2025)
- [Optimizing YOLO27, YOLO26, YOLO11, and YOLOv8 for Fine-Grained Small-Object Detection and Segmentation in Complex Orchard Environments](https://mlsanity.com/paper/2608.23636/) (2026)
- [Selective Cotton Boll Localization for Robotic Harvesting: Evaluation of Deep Learning Vision Models Under Field Conditions](https://mlsanity.com/paper/2609.19592/) (2026)
- [Automated Detection and Encoding of Cartographic Signs in Historical Maps](https://mlsanity.com/paper/2610.13224/) (2026)

## Abstract (arXiv)

Instance segmentation is an important image processing operation for agricultural automation, providing precise delineation of individual objects within images and enabling tasks such as selective harvesting and precision pruning. This study compares the one stage YOLOv8 model with the two stage Mask R CNN model for instance segmentation under varying orchard conditions across two datasets. Dataset 1, collected in the dormant season, contains images of apple trees without foliage and was used to train multi object segmentation models delineating branches and trunks. Dataset 2, collected in the early growing season, includes canopy images with green foliage and immature apples and was used to train single object segmentation models delineating fruitlets. Results showed YOLOv8 outperformed Mask R CNN with higher precision and near perfect recall at a confidence threshold of 0.5. For Dataset 1, YOLOv8 achieved precision 0.90 and recall 0.95 compared to 0.81 and 0.81 for Mask R CNN. For Dataset 2, YOLOv8 reached precision 0.93 and recall 0.97 compared to 0.85 and 0.88. Inference times were also lower for YOLOv8, at 10.9 ms and 7.8 ms, versus 15.6 ms and 12.8 ms for Mask R CNN. These findings demonstrate superior accuracy and efficiency of YOLOv8 for real time orchard automation tasks such as robotic harvesting and fruit thinning.