# Find the Fruit: Zero-Shot Sim2Real RL for Occlusion-Aware Plant Manipulation

arXiv: https://arxiv.org/abs/2505.16547 · HTML: https://mlsanity.com/paper/2505.16547/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 30 of 31)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](https://mlsanity.com/paper/2301.04195/) (2023)
- [SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2011.07215/) (2020)
- [When Is Partially Observable Reinforcement Learning Not Scary?](https://mlsanity.com/paper/2204.08967/) (2022)
- [Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulation](https://mlsanity.com/paper/1703.02018/) (2017)
- [Intel RealSense Stereoscopic Depth Cameras](https://mlsanity.com/paper/1705.05548/) (2017)
- [FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding](https://mlsanity.com/paper/2105.03655/) (2021)
- [Multi-Stage Cable Routing through Hierarchical Imitation Learning](https://mlsanity.com/paper/2307.08927/) (2023)
- [skrl: Modular and Flexible Library for Reinforcement Learning](https://mlsanity.com/paper/2202.03825/) (2022)
- [Movement Primitive Diffusion: Learning Gentle Robotic Manipulation of Deformable Objects](https://mlsanity.com/paper/2312.10008/) (2023)
- [Iterative Residual Policy: for Goal-Conditioned Dynamic Manipulation of Deformable Objects](https://mlsanity.com/paper/2203.00663/) (2022)
- [Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation](https://mlsanity.com/paper/2206.02881/) (2022)
- [Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL](https://mlsanity.com/paper/2410.20254/) (2024)
- [The act of remembering: a study in partially observable reinforcement learning](https://mlsanity.com/paper/2010.01753/) (2020)
- [O2RNet: Occluder-Occludee Relational Network for Robust Apple Detection in Clustered Orchard Environments](https://mlsanity.com/paper/2303.04884/) (2023)
- [A Survey on Robotic Manipulation of Deformable Objects: Recent Advances, Open Challenges and New Frontiers](https://mlsanity.com/paper/2312.10419/) (2023)
- [Visual Perception and Modelling in Unstructured Orchard for Apple Harvesting Robots](https://mlsanity.com/paper/1912.12555/) (2019)
- [Robotic Grasping of Fully-Occluded Objects using RF Perception](https://mlsanity.com/paper/2012.15436/) (2020)
- [dPMP-Deep Probabilistic Motion Planning: A use case in Strawberry Picking Robot](https://mlsanity.com/paper/2208.09074/) (2022)
- [Towards Robotic Tree Manipulation: Leveraging Graph Representations](https://mlsanity.com/paper/2311.07479/) (2023)
- [An Empirical Study on the Power of Future Prediction in Partially Observable Environments](https://mlsanity.com/paper/2402.07102/) (2024)
- [Safe Leaf Manipulation for Accurate Shape and Pose Estimation of Occluded Fruits](https://mlsanity.com/paper/2409.17389/) (2024)
- [Push Past Green: Learning to Look Behind Plant Foliage by Moving It](https://mlsanity.com/paper/2307.03175/) (2023)
- [Interactive Perception for Deformable Object Manipulation](https://mlsanity.com/paper/2403.05177/) (2024)
- [Compact robotic gripper with tandem actuation for selective fruit harvesting](https://mlsanity.com/paper/2408.06674/) (2024)
- [Autonomous Robotic Pruning in Orchards and Vineyards: a Review](https://mlsanity.com/paper/2505.07318/) (2025)
- [Enhanced View Planning for Robotic Harvesting: Tackling Occlusions with Imitation Learning](https://mlsanity.com/paper/2503.10334/) (2025)
- [Zero-shot Sim-to-Real Transfer for Reinforcement Learning-based Visual Servoing of Soft Continuum Arms](https://mlsanity.com/paper/2504.16916/) (2025)
- [Self-Supervised Learning for Robotic Leaf Manipulation: A Hybrid Geometric-Neural Approach](https://mlsanity.com/paper/2505.03702/) (2025)

## Built on by (most cited first, 1 of 1)

- [Reinforcement Learning for the Full Strawberry Harvesting Process: Obstacle Separation, Detachment, and Placement](https://mlsanity.com/paper/2607.14708/) (2026)

## Abstract (arXiv)

Autonomous harvesting in the open presents a complex manipulation problem. In most scenarios, an autonomous system has to deal with significant occlusion and require interaction in the presence of large structural uncertainties (every plant is different). Perceptual and modeling uncertainty make design of reliable manipulation controllers for harvesting challenging, resulting in poor performance during deployment. We present a sim2real reinforcement learning (RL) framework for occlusion-aware plant manipulation, where a policy is learned entirely in simulation to reposition stems and leaves to reveal target fruit(s). In our proposed approach, we decouple high-level kinematic planning from low-level compliant control which simplifies the sim2real transfer. This decomposition allows the learned policy to generalize across multiple plants with different stiffness and morphology. In experiments with multiple real-world plant setups, our system achieves up to 86.7% success in exposing target fruits, demonstrating robustness to occlusion variation and structural uncertainty.