# Learning to Represent Action Values as a Hypergraph on the Action Vertices

arXiv: https://arxiv.org/abs/2010.14680 · HTML: https://mlsanity.com/paper/2010.14680/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2020.

## Built on by (most cited first, 23 of 23)

- [Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare](https://mlsanity.com/paper/2305.01738/) (2023)
- [Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning](https://mlsanity.com/paper/2310.11731/) (2023)
- [Learning Long-Term Reward Redistribution via Randomized Return Decomposition](https://mlsanity.com/paper/2111.13485/) (2021)
- [DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems](https://mlsanity.com/paper/2206.00484/) (2022)
- [Solving Continuous Control via Q-learning](https://mlsanity.com/paper/2210.12566/) (2022)
- [Continuous Control with Action Quantization from Demonstrations](https://mlsanity.com/paper/2110.10149/) (2021)
- [Continuous Control with Coarse-to-fine Reinforcement Learning](https://mlsanity.com/paper/2407.07787/) (2024)
- [Automated Reinforcement Learning: An Overview](https://mlsanity.com/paper/2201.05000/) (2022)
- [Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution](https://mlsanity.com/paper/2404.04253/) (2024)
- [No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learning](https://mlsanity.com/paper/2312.06258/) (2023)
- [REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processes](https://mlsanity.com/paper/2401.08850/) (2024)
- [Predicting Long-Term Human Behaviors in Discrete Representations via Physics-Guided Diffusion](https://mlsanity.com/paper/2405.19528/) (2024)
- [Efficient Policy Generation in Multi-Agent Systems via Hypergraph Neural Network](https://mlsanity.com/paper/2203.03265/) (2022)
- [Orchestrated Value Mapping for Reinforcement Learning](https://mlsanity.com/paper/2203.07171/) (2022)
- [Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learning](https://mlsanity.com/paper/2408.00309/) (2024)
- [Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control](https://mlsanity.com/paper/2510.17122/) (2025)
- [Leveraging Factored Action Spaces for Off-Policy Evaluation](https://mlsanity.com/paper/2307.07014/) (2023)
- [Plug-and-Play Benchmarking of Reinforcement Learning Algorithms for Large-Scale Flow Control](https://mlsanity.com/paper/2601.15015/) (2026)
- [Learning in complex action spaces without policy gradients](https://mlsanity.com/paper/2410.06317/) (2024)
- [Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network](https://mlsanity.com/paper/2502.00288/) (2025)
- [Q-function Decomposition with Intervention Semantics with Factored Action Spaces](https://mlsanity.com/paper/2504.21326/) (2025)
- [QSIM: Mitigating Overestimation in Multi-Agent Reinforcement Learning via Action Similarity Weighted Q-Learning](https://mlsanity.com/paper/2602.22786/) (2026)
- [Value-Preserving Action Quantization](https://mlsanity.com/paper/2610.17252/) (2026)

## Abstract (arXiv)

Action-value estimation is a critical component of many reinforcement learning (RL) methods whereby sample complexity relies heavily on how fast a good estimator for action value can be learned. By viewing this problem through the lens of representation learning, good representations of both state and action can facilitate action-value estimation. While advances in deep learning have seamlessly driven progress in learning state representations, given the specificity of the notion of agency to RL, little attention has been paid to learning action representations. We conjecture that leveraging the combinatorial structure of multi-dimensional action spaces is a key ingredient for learning good representations of action. To test this, we set forth the action hypergraph networks framework -- a class of functions for learning action representations in multi-dimensional discrete action spaces with a structural inductive bias. Using this framework we realise an agent class based on a combination with deep Q-networks, which we dub hypergraph Q-networks. We show the effectiveness of our approach on a myriad of domains: illustrative prediction problems under minimal confounding effects, Atari 2600 games, and discretised physical control benchmarks.