mlsanityData to 14 Oct 2026

Grasp Anything: Combining Teacher-Augmented Policy Gradient Learning with Instance Segmentation to Grasp Arbitrary Objects

arXiv 2403.10187 ↗First posted 15 Mar 2024cs.RO cs.AI cs.LGcited by 3

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

Interactive grasping from clutter, akin to human dexterity, is one of the longest-standing problems in robot learning. Challenges stem from the intricacies of visual perception, the demand for precise motor skills, and the complex interplay between the two. In this work, we present Teacher-Augmented Policy Gradient (TAPG), a novel two-stage learning framework that synergizes reinforcement learning and policy distillation. After training a teacher policy to master the motor control based on object pose information, TAPG facilitates guided, yet adaptive, learning of a sensorimotor policy, based on object segmentation. We zero-shot transfer from simulation to a real robot by using Segment Anything Model for promptable object segmentation. Our trained policies adeptly grasp a wide variety of objects from cluttered scenarios in simulation and the real world based on human-understandable prompts. Furthermore, we show robust zero-shot transfer to novel objects. Videos of our experiments are available at urlhttps://maltemosbach.github.io/graspanything.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 23 of its 28 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

23 of 28 references.

References · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Segment Anythingcited by 8,234
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  6. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  9. 2019Learning by Cheatingcited by 383
  10. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  11. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  12. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
11 more
  1. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  2. 2023SegGPT: Segmenting Everything In Contextcited by 194
  3. 2023Segment and Track Anythingcited by 193
  4. 2021A System for General In-Hand Object Re-Orientationcited by 165
  5. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 128
  6. 2022Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapescited by 101
  7. 2019Distilling Policy Distillationcited by 84
  8. 2018Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methodscited by 33
  9. 2023Dexterous Pre-grasp Manipulation for Human-like Functional Categorical Grasping: Deep Reinforcement Learning and Grasp Representationscited by 4
  10. 2022Efficient Representations of Object Geometry for Reinforcement Learning of Interactive Grasping Policiescited by 2
  11. 2023Learning Generalizable Tool Use with Non-rigid Grasp-pose Registrationcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Learning Flow-Guided Registration for RGB-Event Semantic Segmentationcited by 1
  2. 2025Prompt-responsive Object Retrieval with Memory-augmented Student-Teacher Learningcited by 1
  3. 2025SR3D: Unleashing Single-view 3D Reconstruction for Transparent and Specular Object Graspingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Dexterous Functional Pre-Grasp Manipulation with Diffusion Policy
  2. 2025Learning Adaptive Dexterous Grasping from Single Demonstrations
  3. 2026Grasp-Oriented Non-Prehensile Manipulation via Learning a Graspability Field ↗
  4. 2025RoboGrasp: A Universal Grasping Policy for Robust Robotic Control
  5. 2025ClutterDexGrasp: A Sim-to-Real System for General Dexterous Grasping in Cluttered Scenes
  6. 2024GAP-RL: Grasps As Points for RL Towards Dynamic Object Grasping ↗
  7. 2025FLEX: A Framework for Learning Robot-Agnostic Force-based Skills Involving Sustained Contact Object Manipulation
  8. 2025ArticuBot: Learning Universal Articulated Object Manipulation Policy via Large Scale Simulation