# Learning Generalizable Tool-use Skills through Trajectory Generation

arXiv: https://arxiv.org/abs/2310.00156 · HTML: https://mlsanity.com/paper/2310.00156/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 29 Sep 2023.

## Builds on (most cited first, 23 of 23)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation](https://mlsanity.com/paper/1903.06684/) (2019)
- [PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics](https://mlsanity.com/paper/2104.03311/) (2021)
- [An Analysis of SVD for Deep Rotation Estimation](https://mlsanity.com/paper/2006.14616/) (2020)
- [KETO: Learning Keypoint Representations for Tool Manipulation](https://mlsanity.com/paper/1910.11977/) (2019)
- [DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting](https://mlsanity.com/paper/2105.12244/) (2021)
- [Sinkhorn Divergences for Unbalanced Optimal Transport](https://mlsanity.com/paper/1910.12958/) (2019)
- [ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds](https://mlsanity.com/paper/2211.09006/) (2022)
- [Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision](https://mlsanity.com/paper/1806.09266/) (2018)
- [DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools](https://mlsanity.com/paper/2203.17275/) (2022)
- [kPAM 2.0: Feedback Control for Category-Level Robotic Manipulation](https://mlsanity.com/paper/2102.06279/) (2021)
- [Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight](https://mlsanity.com/paper/1904.05538/) (2019)
- [Projective Manifold Gradient Layer for Deep Rotation Regression](https://mlsanity.com/paper/2110.11657/) (2021)
- [A Smooth Representation of Belief over SO(3) for Deep Rotation Learning with Uncertainty](https://mlsanity.com/paper/2006.01031/) (2020)
- [A Modular Robotic Arm Control Stack for Research: Franka-Interface and FrankaPy](https://mlsanity.com/paper/2011.02398/) (2020)
- [VIRDO: Visio-tactile Implicit Representations of Deformable Objects](https://mlsanity.com/paper/2202.00868/) (2022)
- [RoboNinja: Learning an Adaptive Cutting Policy for Multi-Material Objects](https://mlsanity.com/paper/2302.11553/) (2023)
- [GIFT: Generalizable Interaction-aware Functional Tool Affordances without Labels](https://mlsanity.com/paper/2106.14973/) (2021)
- [VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects](https://mlsanity.com/paper/2210.03701/) (2022)
- [Learning Closed-loop Dough Manipulation Using a Differentiable Reset Module](https://mlsanity.com/paper/2207.04638/) (2022)
- [Understanding Physical Effects for Effective Tool-use](https://mlsanity.com/paper/2206.14998/) (2022)

## Built on by (most cited first, 6 of 6)

- [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023)
- [General Flow as Foundation Affordance for Scalable Robot Learning](https://mlsanity.com/paper/2401.11439/) (2024)
- [VLMgineer: Vision Language Models as Robotic Toolsmiths](https://mlsanity.com/paper/2507.12644/) (2025)
- [SCOOP'D: Learning Mixed-Liquid-Solid Scooping via Sim2Real Generative Policy](https://mlsanity.com/paper/2510.11566/) (2025)
- [Designing Tools with Control Confidence](https://mlsanity.com/paper/2510.12630/) (2025)
- [TIER: Trajectory-Invariant Execution Rewards for Multi-Step Tool Composition](https://mlsanity.com/paper/2605.16790/) (2026)

## Abstract (arXiv)

Autonomous systems that efficiently utilize tools can assist humans in completing many common tasks such as cooking and cleaning. However, current systems fall short of matching human-level of intelligence in terms of adapting to novel tools. Prior works based on affordance often make strong assumptions about the environments and cannot scale to more complex, contact-rich tasks. In this work, we tackle this challenge and explore how agents can learn to use previously unseen tools to manipulate deformable objects. We propose to learn a generative model of the tool-use trajectories as a sequence of tool point clouds, which generalizes to different tool shapes. Given any novel tool, we first generate a tool-use trajectory and then optimize the sequence of tool poses to align with the generated trajectory. We train a single model on four different challenging deformable object manipulation tasks, using demonstration data from only one tool per task. The model generalizes to various novel tools, significantly outperforming baselines. We further test our trained policy in the real world with unseen tools, where it achieves the performance comparable to human. Additional materials can be found on our project website: https://sites.google.com/view/toolgen.