# Characterizing the Action-Generalization Gap in Deep Q-Learning

arXiv: https://arxiv.org/abs/2205.05588 · HTML: https://mlsanity.com/paper/2205.05588/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 8 references that are in the map.
- First posted on arXiv 11 May 2022.

## Builds on (most cited first, 4 of 4)

- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Generalization in Deep Learning](https://mlsanity.com/paper/1710.05468/) (2017)
- [A Theory of Abstraction in Reinforcement Learning](https://mlsanity.com/paper/2203.00397/) (2022)

## Built on by (most cited first, 1 of 1)

- [Learning in complex action spaces without policy gradients](https://mlsanity.com/paper/2410.06317/) (2024)

## Abstract (arXiv)

We study the action generalization ability of deep Q-learning in discrete action spaces. Generalization is crucial for efficient reinforcement learning (RL) because it allows agents to use knowledge learned from past experiences on new tasks. But while function approximation provides deep RL agents with a natural way to generalize over state inputs, the same generalization mechanism does not apply to discrete action outputs. And yet, surprisingly, our experiments indicate that Deep Q-Networks (DQN), which use exactly this type of function approximator, are still able to achieve modest action generalization. Our main contribution is twofold: first, we propose a method of evaluating action generalization using expert knowledge of action similarity, and empirically confirm that action generalization leads to faster learning; second, we characterize the action-generalization gap (the difference in learning performance between DQN and the expert) in different domains. We find that DQN can indeed generalize over actions in several simple domains, but that its ability to do so decreases as the action space grows larger.