# Estimator-Coupled Reinforcement Learning for Robust Purely Tactile In-Hand Manipulation

arXiv: https://arxiv.org/abs/2311.04060 · HTML: https://mlsanity.com/paper/2311.04060/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 17 references that are in the map.
- First posted on arXiv 7 Nov 2023.

## Builds on (most cited first, 11 of 11)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality](https://mlsanity.com/paper/2210.13702/) (2022)
- [In-Hand Object Rotation via Rapid Motor Adaptation](https://mlsanity.com/paper/2210.04887/) (2022)
- [Rotating without Seeing: Towards In-hand Dexterity through Touch](https://mlsanity.com/paper/2303.10880/) (2023)
- [D2RL: Deep Dense Architectures in Reinforcement Learning](https://mlsanity.com/paper/2010.09163/) (2020)
- [DexPBT: Scaling up Dexterous Manipulation for Hand-Arm Systems with Population Based Training](https://mlsanity.com/paper/2305.12127/) (2023)
- [Complex In-Hand Manipulation via Compliance-Enabled Finger Gaiting and Multi-Modal Planning](https://mlsanity.com/paper/2201.07928/) (2022)
- [Sampling-based Exploration for Reinforcement Learning of Dexterous Manipulation](https://mlsanity.com/paper/2303.03486/) (2023)
- [Learning Purely Tactile In-Hand Manipulation with a Torque-Controlled Hand](https://mlsanity.com/paper/2204.03698/) (2022)
- [Dextrous Tactile In-Hand Manipulation Using a Modular Reinforcement Learning Architecture](https://mlsanity.com/paper/2303.04705/) (2023)

## Built on by (most cited first, 9 of 9)

- [Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing](https://mlsanity.com/paper/2312.01853/) (2023)
- [Twisting Lids Off with Two Hands](https://mlsanity.com/paper/2403.02338/) (2024)
- [AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch](https://mlsanity.com/paper/2405.07391/) (2024)
- [From Simple to Complex Skills: The Case of In-Hand Object Reorientation](https://mlsanity.com/paper/2501.05439/) (2025)
- [R×R: Rapid eXploration for Reinforcement Learning via Sampling-based Reset Distributions and Imitation Pre-training](https://mlsanity.com/paper/2401.15484/) (2024)
- [Composing Dextrous Grasping and In-hand Manipulation via Scoring with a Reinforcement Learning Critic](https://mlsanity.com/paper/2505.13253/) (2025)
- [Learning Time-Optimal and Speed-Adjustable Tactile In-Hand Manipulation](https://mlsanity.com/paper/2411.13148/) (2024)
- [roto 2.0: The Robot Tactile Olympiad](https://mlsanity.com/paper/2605.21429/) (2026)
- [Estimate, Don't Imitate: Reusing Differentiable State-Based Policies for Visuomotor Control](https://mlsanity.com/paper/2609.34018/) (2026)

## Abstract (arXiv)

This paper identifies and addresses the problems with naively combining (reinforcement) learning-based controllers and state estimators for robotic in-hand manipulation. Specifically, we tackle the challenging task of purely tactile, goal-conditioned, dextrous in-hand reorientation with the hand pointing downwards. Due to the limited sensing available, many control strategies that are feasible in simulation when having full knowledge of the object's state do not allow for accurate state estimation. Hence, separately training the controller and the estimator and combining the two at test time leads to poor performance. We solve this problem by coupling the control policy to the state estimator already during training in simulation. This approach leads to more robust state estimation and overall higher performance on the task while maintaining an interpretability advantage over end-to-end policy learning. With our GPU-accelerated implementation, learning from scratch takes a median training time of only 6.5 hours on a single, low-cost GPU. In simulation experiments with the DLR-Hand II and for four significantly different object shapes, we provide an in-depth analysis of the performance of our approach. We demonstrate the successful sim2real transfer by rotating the four objects to all 24 orientations in the π/2 discretization of SO(3), which has never been achieved for such a diverse set of shapes. Finally, our method can reorient a cube consecutively to nine goals (median), which was beyond the reach of previous methods in this challenging setting.