# Track Any Peppers: Weakly Supervised Sweet Pepper Tracking Using VLMs

arXiv: https://arxiv.org/abs/2411.06702 · HTML: https://mlsanity.com/paper/2411.06702/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 11 Nov 2024.

## Builds on (most cited first, 11 of 11)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [BoT-SORT: Robust Associations Multi-Pedestrian Tracking](https://mlsanity.com/paper/2206.14651/) (2022)
- [Matching Anything by Segmenting Anything](https://mlsanity.com/paper/2406.04221/) (2024)
- [Benchmarking In-the-wild Multimodal Disease Recognition and A Versatile Baseline](https://mlsanity.com/paper/2408.03120/) (2024)
- [PATHoBot: A Robot for Glasshouse Crop Phenotyping and Intervention](https://mlsanity.com/paper/2010.16272/) (2020)
- [PAg-NeRF: Towards fast and efficient end-to-end panoptic 3D representations for agricultural robotics](https://mlsanity.com/paper/2309.05339/) (2023)
- [PlantSeg: A Large-Scale In-the-wild Dataset for Plant Disease Segmentation](https://mlsanity.com/paper/2409.04038/) (2024)
- [Snap and Diagnose: An Advanced Multimodal Retrieval System for Identifying Plant Diseases in the Wild](https://mlsanity.com/paper/2408.14723/) (2024)
- [CF-PRNet: Coarse-to-Fine Prototype Refining Network for Point Cloud Completion and Reconstruction](https://mlsanity.com/paper/2409.08443/) (2024)

## Built on by (most cited first, 4 of 4)

- [Augment to Segment: Tackling Pixel-Level Imbalance in Wheat Disease and Pest Segmentation](https://mlsanity.com/paper/2509.09961/) (2025)
- [Learning Multi-Modal Prototypes for Cross-Domain Few-Shot Object Detection](https://mlsanity.com/paper/2602.18811/) (2026)
- [Benchmarking Vision-Language Models for Microscopic Plant Image Understanding](https://mlsanity.com/paper/2606.22497/) (2026)
- [PS-MOT: Cultivating Instance Awareness from Point Seeds for Multi-Object Tracking](https://mlsanity.com/paper/2606.30476/) (2026)

## Abstract (arXiv)

In the Detection and Multi-Object Tracking of Sweet Peppers Challenge, we present Track Any Peppers (TAP) - a weakly supervised ensemble technique for sweet peppers tracking. TAP leverages the zero-shot detection capabilities of vision-language foundation models like Grounding DINO to automatically generate pseudo-labels for sweet peppers in video sequences with minimal human intervention. These pseudo-labels, refined when necessary, are used to train a YOLOv8 segmentation network. To enhance detection accuracy under challenging conditions, we incorporate pre-processing techniques such as relighting adjustments and apply depth-based filtering during post-inference. For object tracking, we integrate the Matching by Segment Anything (MASA) adapter with the BoT-SORT algorithm. Our approach achieves a HOTA score of 80.4%, MOTA of 66.1%, Recall of 74.0%, and Precision of 90.7%, demonstrating effective tracking of sweet peppers without extensive manual effort. This work highlights the potential of foundation models for efficient and accurate object detection and tracking in agricultural settings.