# Dot-to-Dot: Explainable Hierarchical Reinforcement Learning for Robotic Manipulation

arXiv: https://arxiv.org/abs/1904.06703 · HTML: https://mlsanity.com/paper/1904.06703/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2019.

## Built on by (most cited first, 30 of 35)

- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey](https://mlsanity.com/paper/2108.09003/) (2021)
- [Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation](https://mlsanity.com/paper/2408.01366/) (2024)
- [Concept Learning for Interpretable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.12232/) (2023)
- [A Survey on AI Assurance](https://mlsanity.com/paper/2111.07505/) (2021)
- [Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunities](https://mlsanity.com/paper/2210.11846/) (2022)
- [Explainability in reinforcement learning: perspective and position](https://mlsanity.com/paper/2203.11547/) (2022)
- [Learning to Centralize Dual-Arm Assembly](https://mlsanity.com/paper/2110.04003/) (2021)
- [Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples](https://mlsanity.com/paper/2310.07747/) (2023)
- [Feature-Based Interpretable Reinforcement Learning based on State-Transition Models](https://mlsanity.com/paper/2105.07099/) (2021)
- [Enabling risk-aware Reinforcement Learning for medical interventions through uncertainty decomposition](https://mlsanity.com/paper/2109.07827/) (2021)
- [Reinforcement Learning Your Way: Agent Characterization through Policy Regularization](https://mlsanity.com/paper/2201.10003/) (2022)
- [Learning Robotic Manipulation Skills Using an Adaptive Force-Impedance Action Space](https://mlsanity.com/paper/2110.09904/) (2021)
- [Distilling Reinforcement Learning Policies for Interpretable Robot Locomotion: Gradient Boosting Machines and Symbolic Regression](https://mlsanity.com/paper/2403.14328/) (2024)
- [Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL](https://mlsanity.com/paper/2510.14129/) (2025)
- [ProtoX: Explaining a Reinforcement Learning Agent via Prototyping](https://mlsanity.com/paper/2211.03162/) (2022)
- [ProxFly: Robust Control for Close Proximity Quadcopter Flight via Residual Reinforcement Learning](https://mlsanity.com/paper/2409.13193/) (2024)
- [Bayesian Distributional Policy Gradients](https://mlsanity.com/paper/2103.11265/) (2021)
- [Hierarchical Primitive Composition: Simultaneous Activation of Skills with Inconsistent Action Dimensions in Multiple Hierarchies](https://mlsanity.com/paper/2110.01833/) (2021)
- [A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges](https://mlsanity.com/paper/2211.06665/) (2022)
- [Embodied Navigation with Auxiliary Task of Action Description Prediction](https://mlsanity.com/paper/2510.21809/) (2025)
- [Sim-and-Real Reinforcement Learning for Manipulation: A Consensus-based Approach](https://mlsanity.com/paper/2302.13423/) (2023)
- [Machine Learning Meets Advanced Robotic Manipulation](https://mlsanity.com/paper/2309.12560/) (2023)
- [Interactive Explanations for Reinforcement-Learning Agents](https://mlsanity.com/paper/2504.05393/) (2025)
- [Reinforcement Learning with Intrinsic Affinity for Personalized Prosperity Management](https://mlsanity.com/paper/2204.09218/) (2022)
- [Off-Policy Deep Reinforcement Learning Algorithms for Handling Various Robotic Manipulator Tasks](https://mlsanity.com/paper/2212.05572/) (2022)
- [Requirements for Explainability and Acceptance of Artificial Intelligence in Collaborative Work](https://mlsanity.com/paper/2306.15394/) (2023)
- [Symbolic Explanation of Affinity-Based Reinforcement Learning Agents with Markov Models](https://mlsanity.com/paper/2208.12627/) (2022)
- [Explaining Agent's Decision-making in a Hierarchical Reinforcement Learning Scenario](https://mlsanity.com/paper/2212.06967/) (2022)

## Abstract (arXiv)

Robotic systems are ever more capable of automation and fulfilment of complex tasks, particularly with reliance on recent advances in intelligent systems, deep learning and artificial intelligence. However, as robots and humans come closer in their interactions, the matter of interpretability, or explainability of robot decision-making processes for the human grows in importance. A successful interaction and collaboration will only take place through mutual understanding of underlying representations of the environment and the task at hand. This is currently a challenge in deep learning systems. We present a hierarchical deep reinforcement learning system, consisting of a low-level agent handling the large actions/states space of a robotic system efficiently, by following the directives of a high-level agent which is learning the high-level dynamics of the environment and task. This high-level agent forms a representation of the world and task at hand that is interpretable for a human operator. The method, which we call Dot-to-Dot, is tested on a MuJoCo-based model of the Fetch Robotics Manipulator, as well as a Shadow Hand, to test its performance. Results show efficient learning of complex actions/states spaces by the low-level agent, and an interpretable representation of the task and decision-making process learned by the high-level agent.