# Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning

arXiv: https://arxiv.org/abs/2111.03062 · HTML: https://mlsanity.com/paper/2111.03062/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 4 Nov 2021.

## Builds on (most cited first, 29 of 29)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging](https://mlsanity.com/paper/1904.06830/) (2019)
- [Multi-task Deep Reinforcement Learning with PopArt](https://mlsanity.com/paper/1809.04474/) (2018)
- [State-Only Imitation Learning for Dexterous Manipulation](https://mlsanity.com/paper/2004.04650/) (2020)
- [Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost](https://mlsanity.com/paper/1810.06045/) (2018)
- [Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning](https://mlsanity.com/paper/1807.03146/) (2018)
- [Learning Dexterous Manipulation for a Soft Robotic Hand from Human Demonstration](https://mlsanity.com/paper/1603.06348/) (2016)
- [Learning Dexterous Manipulation Policies from Experience and Imitation](https://mlsanity.com/paper/1611.05095/) (2016)
- [Learning Dexterous Manipulation from Suboptimal Experts](https://mlsanity.com/paper/2010.08587/) (2020)
- [Solving Challenging Dexterous Manipulation Tasks With Trajectory Optimisation and Reinforcement Learning](https://mlsanity.com/paper/2009.05104/) (2020)
- [In-Hand Manipulation via Motion Cones](https://mlsanity.com/paper/1810.00219/) (2018)
- [Geometric In-Hand Regrasp Planning: Alternating Optimization of Finger Gaits and In-Grasp Manipulation](https://mlsanity.com/paper/1804.04292/) (2018)
- [Experimental Validation of Contact Dynamics for In-Hand Manipulation](https://mlsanity.com/paper/1702.07252/) (2017)

## Built on by (most cited first, 30 of 50)

- [Masked Visual Pre-training for Motor Control](https://mlsanity.com/paper/2203.06173/) (2022)
- [RVT: Robotic View Transformer for 3D Object Manipulation](https://mlsanity.com/paper/2306.14896/) (2023)
- [DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality](https://mlsanity.com/paper/2210.13702/) (2022)
- [UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policy](https://mlsanity.com/paper/2303.00938/) (2023)
- [Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning](https://mlsanity.com/paper/2206.08686/) (2022)
- [In-Hand Object Rotation via Rapid Motor Adaptation](https://mlsanity.com/paper/2210.04887/) (2022)
- [D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions](https://mlsanity.com/paper/2112.03028/) (2021)
- [UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learning](https://mlsanity.com/paper/2304.00464/) (2023)
- [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023)
- [General In-Hand Object Rotation with Vision and Touch](https://mlsanity.com/paper/2309.09979/) (2023)
- [DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation](https://mlsanity.com/paper/2211.09423/) (2022)
- [MyoSuite -- A contact-rich simulation suite for musculoskeletal motor control](https://mlsanity.com/paper/2205.13600/) (2022)
- [LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learning](https://mlsanity.com/paper/2309.06440/) (2023)
- [DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects](https://mlsanity.com/paper/2305.05706/) (2023)
- [Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play](https://mlsanity.com/paper/2303.12076/) (2023)
- [Holo-Dex: Teaching Dexterity with Immersive Mixed Reality](https://mlsanity.com/paper/2210.06463/) (2022)
- [Learning Generalizable Dexterous Manipulation from Human Grasp Affordance](https://mlsanity.com/paper/2204.02320/) (2022)
- [Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation](https://mlsanity.com/paper/2309.00987/) (2023)
- [Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrations](https://mlsanity.com/paper/2207.05053/) (2022)
- [RealDex: Towards Human-like Grasping for Robotic Dexterous Hand](https://mlsanity.com/paper/2402.13853/) (2024)
- [Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Clouds](https://mlsanity.com/paper/2210.07442/) (2022)
- [Learning Dexterous Manipulation from Exemplar Object Trajectories and Pre-Grasps](https://mlsanity.com/paper/2209.11221/) (2022)
- [HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation](https://mlsanity.com/paper/2305.03942/) (2023)
- [PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Training](https://mlsanity.com/paper/2209.11133/) (2022)
- [Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners](https://mlsanity.com/paper/2505.23150/) (2025)
- [Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learning](https://mlsanity.com/paper/2203.13733/) (2022)
- [SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation](https://mlsanity.com/paper/2310.16838/) (2023)
- [DEXOP: A Device for Robotic Transfer of Dexterous Human Manipulation](https://mlsanity.com/paper/2509.04441/) (2025)
- [A Universal Semantic-Geometric Representation for Robotic Manipulation](https://mlsanity.com/paper/2306.10474/) (2023)
- [DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulation](https://mlsanity.com/paper/2411.18562/) (2024)

## Abstract (arXiv)

Dexterous manipulation of arbitrary objects, a fundamental daily task for humans, has been a grand challenge for autonomous robotic systems. Although data-driven approaches using reinforcement learning can develop specialist policies that discover behaviors to control a single object, they often exhibit poor generalization to unseen ones. In this work, we show that policies learned by existing reinforcement learning algorithms can in fact be generalist when combined with multi-task learning and a well-chosen object representation. We show that a single generalist policy can perform in-hand manipulation of over 100 geometrically-diverse real-world objects and generalize to new objects with unseen shape or size. Interestingly, we find that multi-task learning with object point cloud representations not only generalizes better but even outperforms the single-object specialist policies on both training as well as held-out test objects. Video results at https://huangwl18.github.io/geometry-dex