# GOATS: Goal Sampling Adaptation for Scooping with Curriculum Reinforcement Learning

arXiv: https://arxiv.org/abs/2303.05193 · HTML: https://mlsanity.com/paper/2303.05193/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 9 Mar 2023.

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2011.07215/) (2020)
- [Unsupervised Control Through Non-Parametric Discriminative Rewards](https://mlsanity.com/paper/1811.11359/) (2018)
- [Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning](https://mlsanity.com/paper/2007.02832/) (2020)
- [Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards](https://mlsanity.com/paper/1911.01417/) (2019)
- [Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks](https://mlsanity.com/paper/2012.03385/) (2020)
- [ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds](https://mlsanity.com/paper/2211.09006/) (2022)
- [Learning compositional models of robot skills for task and motion planning](https://mlsanity.com/paper/2006.06444/) (2020)
- [Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor](https://mlsanity.com/paper/1910.04854/) (2019)
- [Learning Robotic Manipulation through Visual Planning and Acting](https://mlsanity.com/paper/1905.04411/) (2019)
- [Dynamic Cloth Manipulation with Deep Reinforcement Learning](https://mlsanity.com/paper/1910.14475/) (2019)
- [Cloth Funnels: Canonicalized-Alignment for Multi-Purpose Garment Manipulation](https://mlsanity.com/paper/2210.09347/) (2022)
- [Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation](https://mlsanity.com/paper/2210.10195/) (2022)
- [Rethinking Optimization with Differentiable Simulation from a Global Perspective](https://mlsanity.com/paper/2207.00167/) (2022)
- [Offline-Online Learning of Deformation Model for Cable Manipulation with Graph Neural Networks](https://mlsanity.com/paper/2203.15004/) (2022)
- [Learning Robotic Manipulation of Granular Media](https://mlsanity.com/paper/1709.02833/) (2017)
- [Assistive VR Gym: Interactions with Real People to Improve Virtual Assistive Robots](https://mlsanity.com/paper/2007.04959/) (2020)
- [Can I Pour into It? Robot Imagining Open Containability Affordance of Previously Unseen Objects via Physical Simulations](https://mlsanity.com/paper/2008.02321/) (2020)

## Built on by (most cited first, 5 of 5)

- [Offline Goal-Conditioned Reinforcement Learning for Shape Control of Deformable Linear Objects](https://mlsanity.com/paper/2403.10290/) (2024)
- [Understanding Particles From Video: Property Estimation of Granular Materials via Visuo-Haptic Learning](https://mlsanity.com/paper/2412.02119/) (2024)
- [Learning Tool-Aware Adaptive Compliant Control for Autonomous Regolith Excavation](https://mlsanity.com/paper/2509.05475/) (2025)
- [SCOOP'D: Learning Mixed-Liquid-Solid Scooping via Sim2Real Generative Policy](https://mlsanity.com/paper/2510.11566/) (2025)
- [Few-Shot Adaptation to Non-Stationary Environments via Latent Trend Embedding for Robotics](https://mlsanity.com/paper/2603.10373/) (2026)

## Abstract (arXiv)

In this work, we first formulate the problem of robotic water scooping using goal-conditioned reinforcement learning. This task is particularly challenging due to the complex dynamics of fluids and the need to achieve multi-modal goals. The policy is required to successfully reach both position goals and water amount goals, which leads to a large convoluted goal state space. To overcome these challenges, we introduce Goal Sampling Adaptation for Scooping (GOATS), a curriculum reinforcement learning method that can learn an effective and generalizable policy for robot scooping tasks. Specifically, we use a goal-factorized reward formulation and interpolate position goal distributions and amount goal distributions to create curriculum throughout the learning process. As a result, our proposed method can outperform the baselines in simulation and achieves 5.46% and 8.71% amount errors on bowl scooping and bucket scooping tasks, respectively, under 1000 variations of initial water states in the tank and a large goal state space. Besides being effective in simulation environments, our method can efficiently adapt to noisy real-robot water-scooping scenarios with diverse physical configurations and unseen settings, demonstrating superior efficacy and generalizability. The videos of this work are available on our project page: https://sites.google.com/view/goatscooping.