# ToolNet: Using Commonsense Generalization for Predicting Tool Use for Robot Plan Synthesis

arXiv: https://arxiv.org/abs/2006.05478 · HTML: https://mlsanity.com/paper/2006.05478/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2020.

## Built on by (most cited first, 3 of 3)

- [TANGO: Commonsense Generalization in Predicting Tool Interactions for Mobile Manipulators](https://mlsanity.com/paper/2105.04556/) (2021)
- [DADEE: Well-calibrated uncertainty quantification in neural networks for barriers-based robot safety](https://mlsanity.com/paper/2407.00616/) (2024)
- [GoalNet: Inferring Conjunctive Goal Predicates from Human Plan Demonstrations for Robot Instruction Following](https://mlsanity.com/paper/2205.07081/) (2022)

## Abstract (arXiv)

A robot working in a physical environment (like home or factory) needs to learn to use various available tools for accomplishing different tasks, for instance, a mop for cleaning and a tray for carrying objects. The number of possible tools is large and it may not be feasible to demonstrate usage of each individual tool during training. Can a robot learn commonsense knowledge and adapt to novel settings where some known tools are missing, but alternative unseen tools are present? We present a neural model that predicts the best tool from the available objects for achieving a given declarative goal. This model is trained by user demonstrations, which we crowd-source through humans instructing a robot in a physics simulator. This dataset maintains user plans involving multi-step object interactions along with symbolic state changes. Our neural model, ToolNet, combines a graph neural network to encode the current environment state, and goal-conditioned spatial attention to predict the appropriate tool. We find that providing metric and semantic properties of objects, and pre-trained object embeddings derived from a commonsense knowledge repository such as ConceptNet, significantly improves the model's ability to generalize to unseen tools. The model makes accurate and generalizable tool predictions. When compared to a graph neural network baseline, it achieves 14-27% accuracy improvement for predicting known tools from new world scenes, and 44-67% improvement in generalization for novel objects not encountered during training.