# DEFT: Dexterous Fine-Tuning for Real-World Hand Policies

arXiv: https://arxiv.org/abs/2310.19797 · HTML: https://mlsanity.com/paper/2310.19797/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 79 references that are in the map.
- First posted on arXiv 30 Oct 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [Rapid Locomotion via Reinforcement Learning](https://mlsanity.com/paper/2205.02824/) (2022)
- [Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes](https://mlsanity.com/paper/2211.11744/) (2022)
- [Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube](https://mlsanity.com/paper/2202.10448/) (2022)
- [Data-efficient Deep Reinforcement Learning for Dexterous Manipulation](https://mlsanity.com/paper/1704.03073/) (2017)
- [ReSkin: versatile, replaceable, lasting tactile skins](https://mlsanity.com/paper/2111.00071/) (2021)
- [Stochastic Backpropagation and Approximate Inference in Deep Generative Models](https://mlsanity.com/paper/1401.4082/) (2014)
- [Residual Learning from Demonstration: Adapting DMPs for Contact-rich Manipulation](https://mlsanity.com/paper/2008.07682/) (2020)
- [ALAN: Autonomously Exploring Robotic Agents in the Real World](https://mlsanity.com/paper/2302.06604/) (2023)
- [In Defense of the Direct Perception of Affordances](https://mlsanity.com/paper/1505.01085/) (2015)
- [RobotSweater: Scalable, Generalizable, and Customizable Machine-Knitted Tactile Skins for Robots](https://mlsanity.com/paper/2303.02858/) (2023)

## Built on by (most cited first, 17 of 17)

- [Latent Action Pretraining from Videos](https://mlsanity.com/paper/2410.11758/) (2024)
- [DreamGen: Unlocking Generalization in Robot Learning through Video World Models](https://mlsanity.com/paper/2505.12705/) (2025)
- [RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation](https://mlsanity.com/paper/2412.13877/) (2024)
- [FLARE: Robot Learning with Implicit World Modeling](https://mlsanity.com/paper/2505.15659/) (2025)
- [General Flow as Foundation Affordance for Scalable Robot Learning](https://mlsanity.com/paper/2401.11439/) (2024)
- [Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation](https://mlsanity.com/paper/2401.07487/) (2024)
- [HRP: Human Affordances for Robotic Pre-Training](https://mlsanity.com/paper/2407.18911/) (2024)
- [Hand-Object Interaction Pretraining from Videos](https://mlsanity.com/paper/2409.08273/) (2024)
- [Adaptive Mobile Manipulation for Articulated Objects In the Open World](https://mlsanity.com/paper/2401.14403/) (2024)
- [ViPRA: Video Prediction for Robot Actions](https://mlsanity.com/paper/2511.07732/) (2025)
- [ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection](https://mlsanity.com/paper/2405.03666/) (2024)
- [T-Rex: Tactile-Reactive Dexterous Manipulation](https://mlsanity.com/paper/2606.17055/) (2026)
- [MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videos](https://mlsanity.com/paper/2504.06084/) (2025)
- [DemoStart: Demonstration-led auto-curriculum applied to sim-to-real with multi-fingered robots](https://mlsanity.com/paper/2409.06613/) (2024)
- [Robot Learning from Human Videos: A Survey](https://mlsanity.com/paper/2604.27621/) (2026)
- [HandelBot: Real-World Piano Playing via Fast Adaptation of Dexterous Robot Policies](https://mlsanity.com/paper/2603.12243/) (2026)
- [Mana: Dexterous Manipulation of Articulated Tools](https://mlsanity.com/paper/2606.13677/) (2026)

## Abstract (arXiv)

Dexterity is often seen as a cornerstone of complex manipulation. Humans are able to perform a host of skills with their hands, from making food to operating tools. In this paper, we investigate these challenges, especially in the case of soft, deformable objects as well as complex, relatively long-horizon tasks. However, learning such behaviors from scratch can be data inefficient. To circumvent this, we propose a novel approach, DEFT (DExterous Fine-Tuning for Hand Policies), that leverages human-driven priors, which are executed directly in the real world. In order to improve upon these priors, DEFT involves an efficient online optimization procedure. With the integration of human-based learning and online fine-tuning, coupled with a soft robotic hand, DEFT demonstrates success across various tasks, establishing a robust, data-efficient pathway toward general dexterous manipulation. Please see our website at https://dexterous-finetuning.github.io for video results.