# A Digital Twin Framework for Reinforcement Learning with Real-Time Self-Improvement via Human Assistive Teleoperation

arXiv: https://arxiv.org/abs/2406.00732 · HTML: https://mlsanity.com/paper/2406.00732/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 9 of its 33 references that are in the map.
- First posted on arXiv 2 Jun 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 9 of 9)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Learning to Navigate in Cities Without a Map](https://mlsanity.com/paper/1804.00168/) (2018)
- [DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback](https://mlsanity.com/paper/1810.11748/) (2018)
- [Goal-Driven Autonomous Exploration Through Deep Reinforcement Learning](https://mlsanity.com/paper/2103.07119/) (2021)
- [Deep Q-learning from Demonstrations](https://mlsanity.com/paper/1704.03732/) (2017)
- [Uncertainty-Aware DRL for Autonomous Vehicle Crowd Navigation in Shared Space](https://mlsanity.com/paper/2405.13969/) (2024)
- [A Twin Delayed Deep Deterministic Policy Gradient Algorithm for Autonomous Ground Vehicle Navigation via Digital Twin Perception Awareness](https://mlsanity.com/paper/2403.15067/) (2024)

## Abstract (arXiv)

Reinforcement Learning (RL) or Deep Reinforcement Learning (DRL) is a powerful approach to solving Markov Decision Processes (MDPs) when the model of the environment is not known a priori. However, RL models are still faced with challenges such as handling covariate shifts and ensuring the quality of human demonstration. To address these challenges and further advance DRL models, our work develops a human-in-the-loop DRL framework via digital twin that leverages human intelligence after deployment to retrain the DRL model in real time. First, we develop a pre-trained model fully based on learning through trial and error in the simulated environment allowing scalability and automation while eliminating variability and biases that can come from subjective human guidance. Second, instead of deploying the trained model directly on the UGV, we create a digital twin which controls the physical UGV from the virtual environment. Third, to allow continuous learning without catastrophic forgetting, we introduce the ability of the model to self-improve with the help of small human guidance at the start of the retraining. We test the performance of our proposed model in both simulation and real-world environments with both static and dynamic obstacles. The results indicate that our proposed approach not only outperforms the baseline models in terms of reward accumulation but also demonstrates superior training efficiency.