# How to select and use tools? : Active Perception of Target Objects Using Multimodal Deep Learning

arXiv: https://arxiv.org/abs/2106.02445 · HTML: https://mlsanity.com/paper/2106.02445/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 27 references that are in the map.
- First posted on arXiv 4 Jun 2021.

## Builds on (most cited first, 4 of 4)

- [Revisiting Active Perception](https://mlsanity.com/paper/1603.02729/) (2016)
- [Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision](https://mlsanity.com/paper/1806.09266/) (2018)
- [Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight](https://mlsanity.com/paper/1904.05538/) (2019)
- [Learning Robotic Manipulation of Granular Media](https://mlsanity.com/paper/1709.02833/) (2017)

## Built on by (most cited first, 15 of 15)

- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [Imitation Learning for Nonprehensile Manipulation through Self-Supervised Learning Considering Motion Speed](https://mlsanity.com/paper/2206.10289/) (2022)
- [Understanding Physical Effects for Effective Tool-use](https://mlsanity.com/paper/2206.14998/) (2022)
- [Few-Shot Learning of Force-Based Motions From Demonstration Through Pre-training of Haptic Representation](https://mlsanity.com/paper/2309.04640/) (2023)
- [Grasp, See, and Place: Efficient Unknown Object Rearrangement with Policy Structure Prior](https://mlsanity.com/paper/2402.15402/) (2024)
- [Deep Active Visual Attention for Real-time Robot Motion Generation: Emergence of Tool-body Assimilation and Adaptive Tool-use](https://mlsanity.com/paper/2206.14530/) (2022)
- [When the Inference Meets the Explicitness or Why Multimodality Can Make Us Forget About the Perfect Predictor](https://mlsanity.com/paper/2602.18850/) (2026)
- [Optimizing Active Perception for Learning Simultaneous Viewpoint Selection and Manipulation with Diffusion Policy](https://mlsanity.com/paper/2409.14615/) (2024)
- [DeepGraviLens: a Multi-Modal Architecture for Classifying Gravitational Lensing Data](https://mlsanity.com/paper/2205.00701/) (2022)
- [Real-World Cooking Robot System from Recipes Based on Food State Recognition Using Foundation Models and PDDL](https://mlsanity.com/paper/2410.02874/) (2024)
- [Learning Multimodal Attention for Manipulating Deformable Objects with Changing States](https://mlsanity.com/paper/2309.14837/) (2023)
- [Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challenges](https://mlsanity.com/paper/2508.06832/) (2025)
- [Latent Object Characteristics Recognition with Visual to Haptic-Audio Cross-modal Transfer Learning](https://mlsanity.com/paper/2403.10689/) (2024)
- [Flexible Tool Selection through Low-dimensional Attribute Alignment of Vision and Language](https://mlsanity.com/paper/2505.22146/) (2025)
- [Prolonging Tool Life: Learning Skillful Use of General-purpose Tools through Lifespan-guided Reinforcement Learning](https://mlsanity.com/paper/2507.17275/) (2025)

## Abstract (arXiv)

Selection of appropriate tools and use of them when performing daily tasks is a critical function for introducing robots for domestic applications. In previous studies, however, adaptability to target objects was limited, making it difficult to accordingly change tools and adjust actions. To manipulate various objects with tools, robots must both understand tool functions and recognize object characteristics to discern a tool-object-action relation. We focus on active perception using multimodal sensorimotor data while a robot interacts with objects, and allow the robot to recognize their extrinsic and intrinsic characteristics. We construct a deep neural networks (DNN) model that learns to recognize object characteristics, acquires tool-object-action relations, and generates motions for tool selection and handling. As an example tool-use situation, the robot performs an ingredients transfer task, using a turner or ladle to transfer an ingredient from a pot to a bowl. The results confirm that the robot recognizes object characteristics and servings even when the target ingredients are unknown. We also examine the contributions of images, force, and tactile data and show that learning a variety of multimodal information results in rich perception for tool use.