# DT/MARS-CycleGAN: Improved Object Detection for MARS Phenotyping Robot

arXiv: https://arxiv.org/abs/2310.12787 · HTML: https://mlsanity.com/paper/2310.12787/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 73 references that are in the map.
- First posted on arXiv 19 Oct 2023.

## Builds on (most cited first, 19 of 19)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023)
- [Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks](https://mlsanity.com/paper/1612.05424/) (2016)
- [Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection](https://mlsanity.com/paper/1708.01642/) (2017)
- [RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real](https://mlsanity.com/paper/2006.09001/) (2020)
- [RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer](https://mlsanity.com/paper/2011.03148/) (2020)
- [Global Wheat Head Detection (GWHD) dataset: a large and diverse dataset of high resolution RGB labelled images to develop and benchmark wheat head detection methods](https://mlsanity.com/paper/2005.02162/) (2020)
- [UnrealCV: Connecting Computer Vision to Unreal Engine](https://mlsanity.com/paper/1609.01326/) (2016)
- [Grape detection, segmentation and tracking using deep neural networks and three-dimensional association](https://mlsanity.com/paper/1907.11819/) (2019)
- [Temporal Prediction and Evaluation of Brassica Growth in the Field using Conditional Generative Adversarial Networks](https://mlsanity.com/paper/2105.07789/) (2021)
- [ARIGAN: Synthetic Arabidopsis Plants using Generative Adversarial Network](https://mlsanity.com/paper/1709.00938/) (2017)
- [Mask-guided Vision Transformer (MG-ViT) for Few-Shot Learning](https://mlsanity.com/paper/2205.09995/) (2022)
- [Deep-Learning-based Counting Methods, Datasets, and Applications in Agriculture -- A Review](https://mlsanity.com/paper/2303.02632/) (2023)

## Built on by (most cited first, 4 of 4)

- [Large Language Models for Robotics: Opportunities, Challenges, and Perspectives](https://mlsanity.com/paper/2401.04334/) (2024)
- [Large Language Models for Manufacturing](https://mlsanity.com/paper/2410.21418/) (2024)
- [HARP: Human-Assisted Regrouping with Permutation Invariant Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2409.11741/) (2024)
- [Feedback-driven object detection and iterative model improvement](https://mlsanity.com/paper/2411.19835/) (2024)

## Abstract (arXiv)

Robotic crop phenotyping has emerged as a key technology to assess crops' morphological and physiological traits at scale. These phenotypical measurements are essential for developing new crop varieties with the aim of increasing productivity and dealing with environmental challenges such as climate change. However, developing and deploying crop phenotyping robots face many challenges such as complex and variable crop shapes that complicate robotic object detection, dynamic and unstructured environments that baffle robotic control, and real-time computing and managing big data that challenge robotic hardware/software. This work specifically tackles the first challenge by proposing a novel Digital-Twin(DT)MARS-CycleGAN model for image augmentation to improve our Modular Agricultural Robotic System (MARS)'s crop object detection from complex and variable backgrounds. Our core idea is that in addition to the cycle consistency losses in the CycleGAN model, we designed and enforced a new DT-MARS loss in the deep learning model to penalize the inconsistency between real crop images captured by MARS and synthesized images sensed by DT MARS. Therefore, the generated synthesized crop images closely mimic real images in terms of realism, and they are employed to fine-tune object detectors such as YOLOv8. Extensive experiments demonstrated that our new DT/MARS-CycleGAN framework significantly boosts our MARS' crop object/row detector's performance, contributing to the field of robotic crop phenotyping.