# HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation

arXiv: https://arxiv.org/abs/2109.05490 · HTML: https://mlsanity.com/paper/2109.05490/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 12 Sep 2021.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Decoupling Representation Learning from Reinforcement Learning](https://mlsanity.com/paper/2009.08319/) (2020)
- [Reinforcement Learning with Prototypical Representations](https://mlsanity.com/paper/2102.11271/) (2021)
- [PLAS: Latent Action Space for Offline Reinforcement Learning](https://mlsanity.com/paper/2011.07213/) (2020)
- [Learning Action Representations for Reinforcement Learning](https://mlsanity.com/paper/1902.00183/) (2019)
- [Deep Reinforcement Learning in Parameterized Action Space](https://mlsanity.com/paper/1511.04143/) (2015)
- [Reinforcement Learning with Parameterized Actions](https://mlsanity.com/paper/1509.01644/) (2015)
- [Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space](https://mlsanity.com/paper/1810.06394/) (2018)
- [Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space](https://mlsanity.com/paper/1903.01344/) (2019)
- [High-Dimensional Bayesian Optimisation with Variational Autoencoders and Deep Metric Learning](https://mlsanity.com/paper/2106.03609/) (2021)
- [Improving black-box optimization in VAE latent space using decoder uncertainty](https://mlsanity.com/paper/2107.00096/) (2021)
- [Continuous-Discrete Reinforcement Learning for Hybrid Control in Robotics](https://mlsanity.com/paper/2001.00449/) (2020)
- [Discrete and Continuous Action Representation for Practical RL in Video Games](https://mlsanity.com/paper/1912.11077/) (2019)
- [Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction](https://mlsanity.com/paper/2102.10106/) (2021)
- [Dynamics-aware Embeddings](https://mlsanity.com/paper/1908.09357/) (2019)
- [Multi-Pass Q-Networks for Deep Reinforcement Learning with Parameterised Action Spaces](https://mlsanity.com/paper/1905.04388/) (2019)
- [Automatic Web Testing using Curiosity-Driven Reinforcement Learning](https://mlsanity.com/paper/2103.06018/) (2021)
- [Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces](https://mlsanity.com/paper/1903.04959/) (2019)
- [Hierarchical Approaches for Reinforcement Learning in Parameterized Action Space](https://mlsanity.com/paper/1810.09656/) (2018)
- [Continuous Multiagent Control using Collective Behavior Entropy for Large-Scale Home Energy Management](https://mlsanity.com/paper/2005.10000/) (2020)
- [Foresee then Evaluate: Decomposing Value Estimation with Latent Future Prediction](https://mlsanity.com/paper/2103.02225/) (2021)

## Built on by (most cited first, 30 of 35)

- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters](https://mlsanity.com/paper/2309.11351/) (2023)
- [CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Making](https://mlsanity.com/paper/2406.09509/) (2024)
- [PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaboration](https://mlsanity.com/paper/2203.08553/) (2022)
- [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://mlsanity.com/paper/2310.08348/) (2023)
- [Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn](https://mlsanity.com/paper/2409.04792/) (2024)
- [Generative AI for Deep Reinforcement Learning: Framework, Analysis, and Use Cases](https://mlsanity.com/paper/2405.20568/) (2024)
- [Challenges for Reinforcement Learning in Quantum Circuit Design](https://mlsanity.com/paper/2312.11337/) (2023)
- [Model-based Reinforcement Learning for Parameterized Action Spaces](https://mlsanity.com/paper/2404.03037/) (2024)
- [Meta-Learning Parameterized Skills](https://mlsanity.com/paper/2206.03597/) (2022)
- [Discretionary Lane-Change Decision and Control via Parameterized Soft Actor-Critic for Hybrid Action Space](https://mlsanity.com/paper/2402.15790/) (2024)
- [The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning](https://mlsanity.com/paper/2506.13672/) (2025)
- [Reinforcement learning for traffic signal control in hybrid action space](https://mlsanity.com/paper/2211.12956/) (2022)
- [Action Mapping for Reinforcement Learning in Continuous Environments with Constraints](https://mlsanity.com/paper/2412.04327/) (2024)
- [Distributional Reinforcement Learning for Scheduling of Chemical Production Processes](https://mlsanity.com/paper/2203.00636/) (2022)
- [CycLight: learning traffic signal cooperation with a cycle-level strategy](https://mlsanity.com/paper/2401.08121/) (2024)
- [Action Space Reduction Strategies for Reinforcement Learning in Autonomous Driving](https://mlsanity.com/paper/2507.05251/) (2025)
- [Is Exploration or Optimization the Problem for Deep Reinforcement Learning?](https://mlsanity.com/paper/2508.01329/) (2025)
- [Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies](https://mlsanity.com/paper/2508.03194/) (2025)
- [Logic-informed reinforcement learning for cross-domain optimization of large-scale cyber-physical systems](https://mlsanity.com/paper/2511.00806/) (2025)
- [Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients](https://mlsanity.com/paper/2605.14297/) (2026)
- [Mixed-Integer Optimal Control via Reinforcement Learning: A Case Study on Hybrid Electric Vehicle Energy Management](https://mlsanity.com/paper/2305.01461/) (2023)
- [Maneuver Decision-Making Through Proximal Policy Optimization And Monte Carlo Tree Search](https://mlsanity.com/paper/2309.08611/) (2023)
- [Scheduling Drone and Mobile Charger via Hybrid-Action Deep Reinforcement Learning](https://mlsanity.com/paper/2403.10761/) (2024)
- [Learn A Flexible Exploration Model for Parameterized Action Markov Decision Processes](https://mlsanity.com/paper/2501.02774/) (2025)
- [Pose-RFT: Enhancing MLLMs for 3D Pose Generation via Hybrid Action Reinforcement Fine-Tuning](https://mlsanity.com/paper/2508.07804/) (2025)
- [Strategic Communication under Threat: Learning Information Trade-offs in Pursuit-Evasion Games](https://mlsanity.com/paper/2510.07813/) (2025)
- [Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents](https://mlsanity.com/paper/2511.10687/) (2025)
- [Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions](https://mlsanity.com/paper/2512.20831/) (2025)
- [Breaking the Grid: Distance-Guided Reinforcement Learning in Large Discrete Action Spaces](https://mlsanity.com/paper/2602.08616/) (2026)

## Abstract (arXiv)

Discrete-continuous hybrid action space is a natural setting in many practical problems, such as robot control and game AI. However, most previous Reinforcement Learning (RL) works only demonstrate the success in controlling with either discrete or continuous action space, while seldom take into account the hybrid action space. One naive way to address hybrid action RL is to convert the hybrid action space into a unified homogeneous action space by discretization or continualization, so that conventional RL algorithms can be applied. However, this ignores the underlying structure of hybrid action space and also induces the scalability issue and additional approximation difficulties, thus leading to degenerated results. In this paper, we propose Hybrid Action Representation (HyAR) to learn a compact and decodable latent representation space for the original hybrid action space. HyAR constructs the latent space and embeds the dependence between discrete action and continuous parameter via an embedding table and conditional Variantional Auto-Encoder (VAE). To further improve the effectiveness, the action representation is trained to be semantically smooth through unsupervised environmental dynamics prediction. Finally, the agent then learns its policy with conventional DRL algorithms in the learned representation space and interacts with the environment by decoding the hybrid action embeddings to the original action space. We evaluate HyAR in a variety of environments with discrete-continuous action space. The results demonstrate the superiority of HyAR when compared with previous baselines, especially for high-dimensional action spaces.