# Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight

arXiv: https://arxiv.org/abs/1904.05538 · HTML: https://mlsanity.com/paper/1904.05538/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2019.

## Built on by (most cited first, 30 of 34)

- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Ctrl-World: A Controllable Generative World Model for Robot Manipulation](https://mlsanity.com/paper/2510.10125/) (2025)
- [FitVid: Overfitting in Pixel-Level Video Prediction](https://mlsanity.com/paper/2106.13195/) (2021)
- [COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning](https://mlsanity.com/paper/2010.14500/) (2020)
- [Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction](https://mlsanity.com/paper/2103.04174/) (2021)
- [A Workflow for Offline Model-Free Robotic Reinforcement Learning](https://mlsanity.com/paper/2109.10813/) (2021)
- [Conservative Data Sharing for Multi-Task Offline Reinforcement Learning](https://mlsanity.com/paper/2109.08128/) (2021)
- [How to Leverage Unlabeled Data in Offline Reinforcement Learning](https://mlsanity.com/paper/2202.01741/) (2022)
- [DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools](https://mlsanity.com/paper/2203.17275/) (2022)
- [Creative Robot Tool Use with Large Language Models](https://mlsanity.com/paper/2310.13065/) (2023)
- [Leveraging Language for Accelerated Learning of Tool Manipulation](https://mlsanity.com/paper/2206.13074/) (2022)
- [How to select and use tools? : Active Perception of Target Objects Using Multimodal Deep Learning](https://mlsanity.com/paper/2106.02445/) (2021)
- [Gap-Dependent Unsupervised Exploration for Reinforcement Learning](https://mlsanity.com/paper/2108.05439/) (2021)
- [VLMgineer: Vision Language Models as Robotic Toolsmiths](https://mlsanity.com/paper/2507.12644/) (2025)
- [Learning to Design and Use Tools for Robotic Manipulation](https://mlsanity.com/paper/2311.00754/) (2023)
- [Learning Generalizable Tool-use Skills through Trajectory Generation](https://mlsanity.com/paper/2310.00156/) (2023)
- [Situated GAIL: Multitask imitation using task-conditioned adversarial inverse reinforcement learning](https://mlsanity.com/paper/1911.00238/) (2019)
- [Creative Problem Solving in Artificially Intelligent Agents: A Survey and Framework](https://mlsanity.com/paper/2204.10358/) (2022)
- [RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills](https://mlsanity.com/paper/2506.14763/) (2025)
- [Keep it Simple: Data-efficient Learning for Controlling Complex Systems with Simple Models](https://mlsanity.com/paper/2102.02493/) (2021)
- [All You Need is LUV: Unsupervised Collection of Labeled Images using Invisible UV Fluorescent Indicators](https://mlsanity.com/paper/2203.04566/) (2022)
- [Learning Generalizable Tool Use with Non-rigid Grasp-pose Registration](https://mlsanity.com/paper/2307.16499/) (2023)
- [TopoCut: Learning Multi-Step Cutting with Spectral Rewards and Discrete Diffusion Policies](https://mlsanity.com/paper/2509.19712/) (2025)
- [PiL-World: A Chunk-Wise World Model for VLA Policy-in-the-Loop Evaluation](https://mlsanity.com/paper/2606.05773/) (2026)
- [Visuomotor Grasping with World Models for Surgical Robots](https://mlsanity.com/paper/2508.11200/) (2025)
- [CosmosAlign: Adapting a World Foundation Model for Generative Traffic Video Forecasting](https://mlsanity.com/paper/2608.07693/) (2026)
- [Online Tool Selection with Learned Grasp Prediction Models](https://mlsanity.com/paper/2302.07940/) (2023)
- [Flexible Tool Selection through Low-dimensional Attribute Alignment of Vision and Language](https://mlsanity.com/paper/2505.22146/) (2025)

## Abstract (arXiv)

Machine learning techniques have enabled robots to learn narrow, yet complex tasks and also perform broad, yet simple skills with a wide variety of objects. However, learning a model that can both perform complex tasks and generalize to previously unseen objects and goals remains a significant challenge. We study this challenge in the context of "improvisational" tool use: a robot is presented with novel objects and a user-specified goal (e.g., sweep some clutter into the dustpan), and must figure out, using only raw image observations, how to accomplish the goal using the available objects as tools. We approach this problem by training a model with both a visual and physical understanding of multi-object interactions, and develop a sampling-based optimizer that can leverage these interactions to accomplish tasks. We do so by combining diverse demonstration data with self-supervised interaction data, aiming to leverage the interaction data to build generalizable models and the demonstration data to guide the model-based RL planner to solve complex tasks. Our experiments show that our approach can solve a variety of complex tool use tasks from raw pixel inputs, outperforming both imitation learning and self-supervised learning individually. Furthermore, we show that the robot can perceive and use novel objects as tools, including objects that are not conventional tools, while also choosing dynamically to use or not use tools depending on whether or not they are required.