# Learning a Shape-Conditioned Agent for Purely Tactile In-Hand Manipulation of Various Objects

arXiv: https://arxiv.org/abs/2407.18834 · HTML: https://mlsanity.com/paper/2407.18834/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2024.

## Built on by (most cited first, 8 of 8)

- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [DemoStart: Demonstration-led auto-curriculum applied to sim-to-real with multi-fingered robots](https://mlsanity.com/paper/2409.06613/) (2024)
- [DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Model](https://mlsanity.com/paper/2510.08556/) (2025)
- [Composing Dextrous Grasping and In-hand Manipulation via Scoring with a Reinforcement Learning Critic](https://mlsanity.com/paper/2505.13253/) (2025)
- [DeFM: Learning Foundation Representations from Depth for Robotics](https://mlsanity.com/paper/2601.18923/) (2026)
- [Learning Time-Optimal and Speed-Adjustable Tactile In-Hand Manipulation](https://mlsanity.com/paper/2411.13148/) (2024)
- [ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation](https://mlsanity.com/paper/2604.11138/) (2026)
- [The Role of Touch: Towards Optimal Tactile Sensing Distribution in Anthropomorphic Hands for Dexterous In-Hand Manipulation](https://mlsanity.com/paper/2509.14984/) (2025)

## Abstract (arXiv)

Reorienting diverse objects with a multi-fingered hand is a challenging task. Current methods in robotic in-hand manipulation are either object-specific or require permanent supervision of the object state from visual sensors. This is far from human capabilities and from what is needed in real-world applications. In this work, we address this gap by training shape-conditioned agents to reorient diverse objects in hand, relying purely on tactile feedback (via torque and position measurements of the fingers' joints). To achieve this, we propose a learning framework that exploits shape information in a reinforcement learning policy and a learned state estimator. We find that representing 3D shapes by vectors from a fixed set of basis points to the shape's surface, transformed by its predicted 3D pose, is especially helpful for learning dexterous in-hand manipulation. In simulation and real-world experiments, we show the reorientation of many objects with high success rates, on par with state-of-the-art results obtained with specialized single-object agents. Moreover, we show generalization to novel objects, achieving success rates of ~90% even for non-convex shapes.