# SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation

arXiv: https://arxiv.org/abs/2602.16863 · HTML: https://mlsanity.com/paper/2602.16863/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 82 of its 94 references that are in the map.
- First posted on arXiv 18 Feb 2026.

## Builds on (most cited first, 30 of 82)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [Asymmetric Actor Critic for Image-Based Robot Learning](https://mlsanity.com/paper/1710.06542/) (2017)
- [GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators](https://mlsanity.com/paper/2309.13037/) (2023)
- [A System for General In-Hand Object Re-Orientation](https://mlsanity.com/paper/2111.03043/) (2021)
- [DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation](https://mlsanity.com/paper/2403.07788/) (2024)
- [AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System](https://mlsanity.com/paper/2307.04577/) (2023)
- [DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality](https://mlsanity.com/paper/2210.13702/) (2022)
- [BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation](https://mlsanity.com/paper/2403.09227/) (2024)
- [THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation](https://mlsanity.com/paper/2402.08191/) (2024)
- [UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policy](https://mlsanity.com/paper/2303.00938/) (2023)
- [In-Hand Object Rotation via Rapid Motor Adaptation](https://mlsanity.com/paper/2210.04887/) (2022)
- [DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System](https://mlsanity.com/paper/1910.03135/) (2019)
- [UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learning](https://mlsanity.com/paper/2304.00464/) (2023)
- [Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes](https://mlsanity.com/paper/2211.11744/) (2022)
- [Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube](https://mlsanity.com/paper/2202.10448/) (2022)
- [FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learning](https://mlsanity.com/paper/2401.08553/) (2024)
- [Humanoid Policy ~ Human Policy](https://mlsanity.com/paper/2503.13441/) (2025)
- [Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning](https://mlsanity.com/paper/2407.03162/) (2024)
- [Learning Visuotactile Skills with Two Multifingered Hands](https://mlsanity.com/paper/2404.16823/) (2024)

## Built on by (most cited first, 15 of 15)

- [LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition](https://mlsanity.com/paper/2606.11628/) (2026)
- [A Minimalist Retargeting-Guided Reinforcement Learning Recipe for Dexterous Manipulation](https://mlsanity.com/paper/2607.11874/) (2026)
- [UniCross: Unified Cross-Skill Dexterous Manipulation Synthesis](https://mlsanity.com/paper/2607.28198/) (2026)
- [ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning](https://mlsanity.com/paper/2608.19182/) (2026)
- [Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?](https://mlsanity.com/paper/2606.26428/) (2026)
- [Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experience](https://mlsanity.com/paper/2606.27475/) (2026)
- [Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation](https://mlsanity.com/paper/2609.14878/) (2026)
- [HandelBot: Real-World Piano Playing via Fast Adaptation of Dexterous Robot Policies](https://mlsanity.com/paper/2603.12243/) (2026)
- [Mana: Dexterous Manipulation of Articulated Tools](https://mlsanity.com/paper/2606.13677/) (2026)
- [ORCA: A Platform for Open-Source Dexterity Research](https://mlsanity.com/paper/2606.14561/) (2026)
- [From Pixel to Poses: Object-centric Tool Manipulation Learning from Human Demonstrations](https://mlsanity.com/paper/2609.35375/) (2026)
- [X-Reset: Scaling Object-Centric Reinforcement Learning via Cross-Embodiment Resets](https://mlsanity.com/paper/2609.35715/) (2026)
- [GOTT: Object-centric Dexterous Manipulation with a Reusable Cross-Embodiment Primitive](https://mlsanity.com/paper/2610.03861/) (2026)
- [DexFLEX: Contact-Aware Foundation Controller for Command-Guided Dexterity](https://mlsanity.com/paper/2610.14372/) (2026)
- [DexCast: Learning Real-World Dexterous Control from Human Videos](https://mlsanity.com/paper/2610.14691/) (2026)

## Abstract (arXiv)

The ability to manipulate tools significantly expands the set of tasks a robot can perform. Yet, tool manipulation represents a challenging class of dexterity, requiring grasping thin objects, in-hand object rotations, and forceful interactions. Since collecting teleoperation data for these behaviors is challenging, sim-to-real reinforcement learning (RL) is a promising alternative. However, prior approaches typically require substantial engineering effort to model objects and tune reward functions for each task. In this work, we propose SimToolReal, taking a step towards generalizing sim-to-real RL policies for tool manipulation. Instead of focusing on a single object and task, we procedurally generate a large variety of tool-like object primitives in simulation and train a single RL policy with the universal goal of manipulating each object to random goal poses. This approach enables SimToolReal to perform general dexterous tool manipulation at test-time without any object or task-specific training. We demonstrate that SimToolReal outperforms prior retargeting and fixed-grasp methods by 37% while matching the performance of specialist RL policies trained on specific target objects and tasks. Finally, we show that SimToolReal generalizes across a diverse set of everyday tools, achieving strong zero-shot performance over 120 real-world rollouts spanning 24 tasks, 12 object instances, and 6 tool categories.