# Learning how to Interact with a Complex Interface using Hierarchical Reinforcement Learning

arXiv: https://arxiv.org/abs/2204.10374 · HTML: https://mlsanity.com/paper/2204.10374/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 26 references that are in the map.
- First posted on arXiv 21 Apr 2022.

## Builds on (most cited first, 9 of 9)

- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Acme: A Research Framework for Distributed Reinforcement Learning](https://mlsanity.com/paper/2006.00979/) (2020)
- [Language as an Abstraction for Hierarchical Deep Reinforcement Learning](https://mlsanity.com/paper/1906.07343/) (2019)
- [A Laplacian Framework for Option Discovery in Reinforcement Learning](https://mlsanity.com/paper/1703.00956/) (2017)
- [AndroidEnv: A Reinforcement Learning Platform for Android](https://mlsanity.com/paper/2105.13231/) (2021)
- [Empowerment -- an Introduction](https://mlsanity.com/paper/1310.1863/) (2013)
- [Temporal-Difference Networks](https://mlsanity.com/paper/1504.05539/) (2015)

## Built on by (most cited first, 3 of 3)

- [Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference](https://mlsanity.com/paper/2412.14355/) (2024)
- [Hierarchical Reinforcement Learning Framework for Adaptive Walking Control Using General Value Functions of Lower-Limb Sensor Signals](https://mlsanity.com/paper/2507.16983/) (2025)
- [Towards General Computer Control with Hierarchical Agents and Multi-Level Action Spaces](https://mlsanity.com/paper/2509.18230/) (2025)

## Abstract (arXiv)

Hierarchical Reinforcement Learning (HRL) allows interactive agents to decompose complex problems into a hierarchy of sub-tasks. Higher-level tasks can invoke the solutions of lower-level tasks as if they were primitive actions. In this work, we study the utility of hierarchical decompositions for learning an appropriate way to interact with a complex interface. Specifically, we train HRL agents that can interface with applications in a simulated Android device. We introduce a Hierarchical Distributed Deep Reinforcement Learning architecture that learns (1) subtasks corresponding to simple finger gestures, and (2) how to combine these gestures to solve several Android tasks. Our approach relies on goal conditioning and can be used more generally to convert any base RL agent into an HRL agent. We use the AndroidEnv environment to evaluate our approach. For the experiments, the HRL agent uses a distributed version of the popular DQN algorithm to train different components of the hierarchy. While the native action space is completely intractable for simple DQN agents, our architecture can be used to establish an effective way to interact with different tasks, significantly improving the performance of the same DQN agent over different levels of abstraction.