# Discrete and Continuous Action Representation for Practical RL in Video Games

arXiv: https://arxiv.org/abs/1912.11077 · HTML: https://mlsanity.com/paper/1912.11077/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2019.

## Built on by (most cited first, 30 of 39)

- [Hierarchical Skills for Efficient Exploration](https://mlsanity.com/paper/2110.10809/) (2021)
- [HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation](https://mlsanity.com/paper/2109.05490/) (2021)
- [Benchmarking End-to-End Behavioural Cloning on Video Games](https://mlsanity.com/paper/2004.00981/) (2020)
- [TAAC: Temporally Abstract Actor-Critic for Continuous Control](https://mlsanity.com/paper/2104.06521/) (2021)
- [Monte Carlo Tree Search based Hybrid Optimization of Variational Quantum Circuits](https://mlsanity.com/paper/2203.16707/) (2022)
- [Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learning](https://mlsanity.com/paper/2311.15341/) (2023)
- [Identifying optimal cycles in quantum thermal machines with reinforcement-learning](https://mlsanity.com/paper/2108.13525/) (2021)
- [A Hybrid Approach for Learning to Shift and Grasp with Elaborate Motion Primitives](https://mlsanity.com/paper/2111.01510/) (2021)
- [Discretionary Lane-Change Decision and Control via Parameterized Soft Actor-Critic for Hybrid Action Space](https://mlsanity.com/paper/2402.15790/) (2024)
- [Policy Fusion for Adaptive and Customizable Reinforcement Learning Agents](https://mlsanity.com/paper/2104.10610/) (2021)
- [Design Strategy Network: A deep hierarchical framework to represent generative design strategies in complex action spaces](https://mlsanity.com/paper/2110.03760/) (2021)
- [Robust Imitation Learning for Automated Game Testing](https://mlsanity.com/paper/2401.04572/) (2024)
- [Soft Actor-Critic With Integer Actions](https://mlsanity.com/paper/2109.08512/) (2021)
- [A learning-based approach to stochastic optimal control under reach-avoid constraint](https://mlsanity.com/paper/2412.16561/) (2024)
- [Advancing Safe Mechanical Ventilation Using Offline RL With Hybrid Actions and Clinically Aligned Rewards](https://mlsanity.com/paper/2506.14375/) (2025)
- [Investigation of Factorized Optical Flows as Mid-Level Representations](https://mlsanity.com/paper/2203.04927/) (2022)
- [GenAI-Net: A Generative AI Framework for Automated Biomolecular Network Design](https://mlsanity.com/paper/2601.17582/) (2026)
- [Model-free optimization of power/efficiency tradeoffs in quantum thermal machines using reinforcement learning](https://mlsanity.com/paper/2204.04785/) (2022)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [Virtual Guidance as a Mid-level Representation for Navigation with Augmented Reality](https://mlsanity.com/paper/2303.02731/) (2023)
- [Visual Forecasting as a Mid-level Representation for Avoidance](https://mlsanity.com/paper/2310.07724/) (2023)
- [Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policies](https://mlsanity.com/paper/2405.01198/) (2024)
- [Artificially intelligent Maxwell's demon for optimal control of open quantum systems](https://mlsanity.com/paper/2408.15328/) (2024)
- [Hybrid TD3: Overestimation Bias Analysis and Stable Policy Optimization for Hybrid Action Space](https://mlsanity.com/paper/2603.01302/) (2026)
- [Dmsh: A Multi-Agent Reinforcement Learning Framework for All-Quad Mesh Generation](https://mlsanity.com/paper/2606.10601/) (2026)
- [Building Decision Forest via Deep Reinforcement Learning](https://mlsanity.com/paper/2204.00306/) (2022)
- [Deep Multi-Agent Reinforcement Learning with Hybrid Action Spaces based on Maximum Entropy](https://mlsanity.com/paper/2206.05108/) (2022)
- [A further exploration of deep Multi-Agent Reinforcement Learning with Hybrid Action Space](https://mlsanity.com/paper/2208.14447/) (2022)
- [Computational Discovery of Energy-Efficient Heat Treatment for Microstructure Design using Deep Reinforcement Learning](https://mlsanity.com/paper/2209.11259/) (2022)
- [Position control of an acoustic cavitation bubble by reinforcement learning](https://mlsanity.com/paper/2312.05674/) (2023)

## Abstract (arXiv)

While most current research in Reinforcement Learning (RL) focuses on improving the performance of the algorithms in controlled environments, the use of RL under constraints like those met in the video game industry is rarely studied. Operating under such constraints, we propose Hybrid SAC, an extension of the Soft Actor-Critic algorithm able to handle discrete, continuous and parameterized actions in a principled way. We show that Hybrid SAC can successfully solve a highspeed driving task in one of our games, and is competitive with the state-of-the-art on parameterized actions benchmark tasks. We also explore the impact of using normalizing flows to enrich the expressiveness of the policy at minimal computational cost, and identify a potential undesired effect of SAC when used with normalizing flows, that may be addressed by optimizing a different objective.