# Scaling All-Goals Updates in Reinforcement Learning Using Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1810.02927 · HTML: https://mlsanity.com/paper/1810.02927/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2018.

## Built on by (most cited first, 1 of 1)

- [Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learning](https://mlsanity.com/paper/2110.12985/) (2021)

## Abstract (arXiv)

Being able to reach any desired location in the environment can be a valuable asset for an agent. Learning a policy to navigate between all pairs of states individually is often not feasible. An all-goals updating algorithm uses each transition to learn Q-values towards all goals simultaneously and off-policy. However the expensive numerous updates in parallel limited the approach to small tabular cases so far. To tackle this problem we propose to use convolutional network architectures to generate Q-values and updates for a large number of goals at once. We demonstrate the accuracy and generalization qualities of the proposed method on randomly generated mazes and Sokoban puzzles. In the case of on-screen goal coordinates the resulting mapping from frames to distance-maps directly informs the agent about which places are reachable and in how many steps. As an example of application we show that replacing the random actions in epsilon-greedy exploration by several actions towards feasible goals generates better exploratory trajectories on Montezuma's Revenge and Super Mario All-Stars games.