# DiAReL: Reinforcement Learning with Disturbance Awareness for Robust Sim2Real Policy Transfer in Robot Control

arXiv: https://arxiv.org/abs/2306.09010 · HTML: https://mlsanity.com/paper/2306.09010/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 64 references that are in the map.
- First posted on arXiv 15 Jun 2023.

## Builds on (most cited first, 30 of 41)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [Action Robust Reinforcement Learning and Applications in Continuous Control](https://mlsanity.com/paper/1901.09184/) (2019)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping](https://mlsanity.com/paper/1709.07857/) (2017)
- [Preparing for the Unknown: Learning a Universal Policy with Online System Identification](https://mlsanity.com/paper/1702.02453/) (2017)
- [EPOpt: Learning Robust Neural Network Policies Using Model Ensembles](https://mlsanity.com/paper/1610.01283/) (2016)
- [Assessing Generalization in Deep Reinforcement Learning](https://mlsanity.com/paper/1810.12282/) (2018)
- [Sim-to-Real Robot Learning from Pixels with Progressive Nets](https://mlsanity.com/paper/1610.04286/) (2016)
- [Reinforcement Learning with Random Delays](https://mlsanity.com/paper/2010.02966/) (2020)
- [Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model](https://mlsanity.com/paper/1610.03518/) (2016)
- [Wasserstein Robust Reinforcement Learning](https://mlsanity.com/paper/1907.13196/) (2019)
- [Real-Time Reinforcement Learning](https://mlsanity.com/paper/1911.04448/) (2019)
- [Learning Contact-Rich Manipulation Skills with Guided Policy Search](https://mlsanity.com/paper/1501.05611/) (2015)
- [Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulation](https://mlsanity.com/paper/1710.06422/) (2017)
- [Delay-Aware Model-Based Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/2005.05440/) (2020)
- [Setting up a Reinforcement Learning Task with a Real-World Robot](https://mlsanity.com/paper/1803.07067/) (2018)

## Built on by (most cited first, 7 of 7)

- [Continual Domain Randomization](https://mlsanity.com/paper/2403.12193/) (2024)
- [Safe Continual Domain Adaptation after Sim2Real Transfer of Reinforcement Learning Policies in Robotics](https://mlsanity.com/paper/2503.10949/) (2025)
- [Reinforcement Learning for Control Systems with Time Delays: A Comprehensive Survey](https://mlsanity.com/paper/2602.00399/) (2026)
- [Online Adaptive Reinforcement Learning with Echo State Networks for Non-Stationary Dynamics](https://mlsanity.com/paper/2602.06326/) (2026)
- [Reinforcement Learning on Cost-Constrained Quadrupedal Hardware](https://mlsanity.com/paper/2607.26434/) (2026)
- [Enhancing Sim2Real Transfer for Torque-Controlled Robots through Real2Sim Dynamics Estimation and Reinforcement Learning](https://mlsanity.com/paper/2608.22629/) (2026)
- [Stability-aware Residual Reinforcement Learning Framework for Robotic Manipulator Disturbance Compensation](https://mlsanity.com/paper/2609.21307/) (2026)

## Abstract (arXiv)

Delayed Markov decision processes (DMDPs) fulfill the Markov property by augmenting the state space of agents with a finite time window of recently committed actions. In reliance on these state augmentations, delay-resolved reinforcement learning algorithms train policies to learn optimal interactions with environments featuring observation or action delays. Although such methods can be directly trained on the real robots, due to sample inefficiency, limited resources, or safety constraints, a common approach is to transfer models trained in simulation to the physical robot. However, robotic simulations rely on approximated models of the physical systems, which hinders the sim2real transfer. In this work, we consider various uncertainties in modeling the robot or environment dynamics as unknown intrinsic disturbances applied to the system input. We introduce the disturbance-augmented Markov decision process (DAMDP) in delayed settings as a novel representation to incorporate disturbance estimation in training on-policy reinforcement learning algorithms. The proposed method is validated across several metrics on learning robotic reaching and pushing tasks and compared with disturbance-unaware baselines. The results show that the disturbance-augmented models can achieve higher stabilization and robustness in the control response, which in turn improves the prospects of successful sim2real transfer.