# Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision

arXiv: https://arxiv.org/abs/1806.09266 · HTML: https://mlsanity.com/paper/1806.09266/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2018.

## Built on by (most cited first, 30 of 52)

- [Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes](https://mlsanity.com/paper/2103.14127/) (2021)
- [ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations](https://mlsanity.com/paper/2107.14483/) (2021)
- [OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction](https://mlsanity.com/paper/2203.15709/) (2022)
- [Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations](https://mlsanity.com/paper/2104.01542/) (2021)
- [ManipulaTHOR: A Framework for Visual Object Manipulation](https://mlsanity.com/paper/2104.11213/) (2021)
- [VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects](https://mlsanity.com/paper/2106.14440/) (2021)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation](https://mlsanity.com/paper/2109.09163/) (2021)
- [DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools](https://mlsanity.com/paper/2203.17275/) (2022)
- [Creative Robot Tool Use with Large Language Models](https://mlsanity.com/paper/2310.13065/) (2023)
- [M2T2: Multi-Task Masked Transformer for Object-centric Pick and Place](https://mlsanity.com/paper/2311.00926/) (2023)
- [One-Shot Object Affordance Detection in the Wild](https://mlsanity.com/paper/2108.03658/) (2021)
- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Training](https://mlsanity.com/paper/2507.13097/) (2025)
- [Leveraging Language for Accelerated Learning of Tool Manipulation](https://mlsanity.com/paper/2206.13074/) (2022)
- [Learning Visual Affordance Grounding from Demonstration Videos](https://mlsanity.com/paper/2108.05675/) (2021)
- [A Joint Network for Grasp Detection Conditioned on Natural Language Commands](https://mlsanity.com/paper/2104.00492/) (2021)
- [How to select and use tools? : Active Perception of Target Objects Using Multimodal Deep Learning](https://mlsanity.com/paper/2106.02445/) (2021)
- [ShapeGrasp: Zero-Shot Task-Oriented Grasping with Large Language Models through Geometric Decomposition](https://mlsanity.com/paper/2403.18062/) (2024)
- [FingerSLAM: Closed-loop Unknown Object Localization and Reconstruction from Visuo-tactile Feedback](https://mlsanity.com/paper/2303.07997/) (2023)
- [Collision-Aware Target-Driven Object Grasping in Constrained Environments](https://mlsanity.com/paper/2104.00776/) (2021)
- [VLMgineer: Vision Language Models as Robotic Toolsmiths](https://mlsanity.com/paper/2507.12644/) (2025)
- [Learning Reusable Manipulation Strategies](https://mlsanity.com/paper/2311.03293/) (2023)
- [One-Shot Manipulation Strategy Learning by Making Contact Analogies](https://mlsanity.com/paper/2411.09627/) (2024)
- [Learning to Design and Use Tools for Robotic Manipulation](https://mlsanity.com/paper/2311.00754/) (2023)
- [Scene Editing as Teleoperation: A Case Study in 6DoF Kit Assembly](https://mlsanity.com/paper/2110.04450/) (2021)
- [Learning Generalizable Tool-use Skills through Trajectory Generation](https://mlsanity.com/paper/2310.00156/) (2023)
- [Discovering Generalizable Skills via Automated Generation of Diverse Tasks](https://mlsanity.com/paper/2106.13935/) (2021)
- [MaxMI: A Maximal Mutual Information Criterion for Manipulation Concept Discovery](https://mlsanity.com/paper/2407.15086/) (2024)
- [Task-Oriented 6-DoF Grasp Pose Detection in Clutters](https://mlsanity.com/paper/2502.16976/) (2025)

## Abstract (arXiv)

Tool manipulation is vital for facilitating robots to complete challenging task goals. It requires reasoning about the desired effect of the task and thus properly grasping and manipulating the tool to achieve the task. Task-agnostic grasping optimizes for grasp robustness while ignoring crucial task-specific constraints. In this paper, we propose the Task-Oriented Grasping Network (TOG-Net) to jointly optimize both task-oriented grasping of a tool and the manipulation policy for that tool. The training process of the model is based on large-scale simulated self-supervision with procedurally generated tool objects. We perform both simulated and real-world experiments on two tool-based manipulation tasks: sweeping and hammering. Our model achieves overall 71.1% task success rate for sweeping and 80.0% task success rate for hammering. Supplementary material is available at: bit.ly/task-oriented-grasp