# MetaFruit Meets Foundation Models: Leveraging a Comprehensive Multi-Fruit Dataset for Advancing Agricultural Foundation Models

arXiv: https://arxiv.org/abs/2407.04711 · HTML: https://mlsanity.com/paper/2407.04711/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 82 references that are in the map.
- First posted on arXiv 14 May 2024.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [Recent Advances in Open Set Recognition: A Survey](https://mlsanity.com/paper/1811.08581/) (2018)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [SqueezeLLM: Dense-and-Sparse Quantization](https://mlsanity.com/paper/2306.07629/) (2023)
- [A Survey on Model Compression for Large Language Models](https://mlsanity.com/paper/2308.07633/) (2023)
- [Foundation Models in Robotics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2312.07843/) (2023)
- [RTMDet: An Empirical Study of Designing Real-Time Object Detectors](https://mlsanity.com/paper/2212.07784/) (2022)
- [FCOS: A simple and strong anchor-free object detector](https://mlsanity.com/paper/2006.09214/) (2020)
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Deep learning in agriculture: A survey](https://mlsanity.com/paper/1807.11809/) (2018)
- [MinneApple: A Benchmark Dataset for Apple Detection and Segmentation](https://mlsanity.com/paper/1909.06441/) (2019)
- [Deep Fruit Detection in Orchards](https://mlsanity.com/paper/1610.03677/) (2016)
- [Label-Efficient Learning in Agriculture: A Comprehensive Review](https://mlsanity.com/paper/2305.14691/) (2023)
- [Leaf Only SAM: A Segment Anything Pipeline for Zero-Shot Automated Leaf Segmentation](https://mlsanity.com/paper/2305.09418/) (2023)
- [O2RNet: Occluder-Occludee Relational Network for Robust Apple Detection in Clustered Orchard Environments](https://mlsanity.com/paper/2303.04884/) (2023)

## Built on by (most cited first, 4 of 4)

- [Geo-OLM: Enabling Sustainable Earth Observation Studies with Cost-Efficient Open Language Models & State-Driven Workflows](https://mlsanity.com/paper/2504.04319/) (2025)
- [Learn from Foundation Model: Fruit Detection Model without Manual Annotation](https://mlsanity.com/paper/2411.16196/) (2024)
- [VLLFL: A Vision-Language Model Based Lightweight Federated Learning Framework for Smart Agriculture](https://mlsanity.com/paper/2504.13365/) (2025)
- [FedReplay: A Feature Replay Assisted Federated Transfer Learning Framework for Efficient and Privacy-Preserving Smart Agriculture](https://mlsanity.com/paper/2511.00269/) (2025)

## Abstract (arXiv)

Fruit harvesting poses a significant labor and financial burden for the industry, highlighting the critical need for advancements in robotic harvesting solutions. Machine vision-based fruit detection has been recognized as a crucial component for robust identification of fruits to guide robotic manipulation. Despite considerable progress in leveraging deep learning and machine learning techniques for fruit detection, a common shortfall is the inability to swiftly extend the developed models across different orchards and/or various fruit species. Additionally, the limited availability of pertinent data further compounds these challenges. In this work, we introduce MetaFruit, the largest publicly available multi-class fruit dataset, comprising 4,248 images and 248,015 manually labeled instances across diverse U.S. orchards. Furthermore, this study proposes an innovative open-set fruit detection system leveraging advanced Vision Foundation Models (VFMs) for fruit detection that can adeptly identify a wide array of fruit types under varying orchard conditions. This system not only demonstrates remarkable adaptability in learning from minimal data through few-shot learning but also shows the ability to interpret human instructions for subtle detection tasks. The performance of the developed foundation model is comprehensively evaluated using several metrics, which outperforms the existing state-of-the-art algorithms in both our MetaFruit dataset and other open-sourced fruit datasets, thereby setting a new benchmark in the field of agricultural technology and robotic harvesting. The MetaFruit dataset and detection framework are open-sourced to foster future research in vision-based fruit harvesting, marking a significant stride toward addressing the urgent needs of the agricultural sector.