# Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space

arXiv: https://arxiv.org/abs/1810.06394 · HTML: https://mlsanity.com/paper/1810.06394/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2018.

## Built on by (most cited first, 30 of 58)

- [Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks](https://mlsanity.com/paper/2110.03655/) (2021)
- [Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives](https://mlsanity.com/paper/2110.15360/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Deep Imitation Learning for Bimanual Robotic Manipulation](https://mlsanity.com/paper/2010.05134/) (2020)
- [HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation](https://mlsanity.com/paper/2109.05490/) (2021)
- [HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation](https://mlsanity.com/paper/2305.03942/) (2023)
- [ReLAX: Reinforcement Learning Agent eXplainer for Arbitrary Predictive Models](https://mlsanity.com/paper/2110.11960/) (2021)
- [Survey of Recent Multi-Agent Reinforcement Learning Algorithms Utilizing Centralized Training](https://mlsanity.com/paper/2107.14316/) (2021)
- [Automated Synthesis of Steady-State Continuous Processes using Reinforcement Learning](https://mlsanity.com/paper/2101.04422/) (2021)
- [RELIEF: Reinforcement Learning Empowered Graph Feature Prompt Tuning](https://mlsanity.com/paper/2408.03195/) (2024)
- [Model-based Reinforcement Learning for Parameterized Action Spaces](https://mlsanity.com/paper/2404.03037/) (2024)
- [Meta-Learning Parameterized Skills](https://mlsanity.com/paper/2206.03597/) (2022)
- [Discretionary Lane-Change Decision and Control via Parameterized Soft Actor-Critic for Hybrid Action Space](https://mlsanity.com/paper/2402.15790/) (2024)
- [HACMan++: Spatially-Grounded Motion Primitives for Manipulation](https://mlsanity.com/paper/2407.08585/) (2024)
- [Reinforcement learning for traffic signal control in hybrid action space](https://mlsanity.com/paper/2211.12956/) (2022)
- [Design Strategy Network: A deep hierarchical framework to represent generative design strategies in complex action spaces](https://mlsanity.com/paper/2110.03760/) (2021)
- [Counter-Empirical Attacking based on Adversarial Reinforcement Learning for Time-Relevant Scoring System](https://mlsanity.com/paper/2311.05144/) (2023)
- [Learning-Based Client Selection for Federated Learning Services Over Wireless Networks with Constrained Monetary Budgets](https://mlsanity.com/paper/2208.04322/) (2022)
- [Optimal Scheduling in IoT-Driven Smart Isolated Microgrids Based on Deep Reinforcement Learning](https://mlsanity.com/paper/2305.00127/) (2023)
- [A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective](https://mlsanity.com/paper/2503.23650/) (2025)
- [Multi-Functional RIS-Enabled in SAGIN for IoT: A Hybrid Deep Reinforcement Learning Approach with Compressed Twin-Models](https://mlsanity.com/paper/2507.16204/) (2025)
- [Collaborative Intelligent Reflecting Surface Networks with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2203.14152/) (2022)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [CycLight: learning traffic signal cooperation with a cycle-level strategy](https://mlsanity.com/paper/2401.08121/) (2024)
- [Constrained Optimal Fuel Consumption of HEV: A Constrained Reinforcement Learning Approach](https://mlsanity.com/paper/2403.07503/) (2024)
- [Detecting Continuous Integration Skip : A Reinforcement Learning-based Approach](https://mlsanity.com/paper/2405.09657/) (2024)
- [Learning Parameterized Skills from Demonstrations](https://mlsanity.com/paper/2510.24095/) (2025)
- [Logic-informed reinforcement learning for cross-domain optimization of large-scale cyber-physical systems](https://mlsanity.com/paper/2511.00806/) (2025)
- [Parametrized Sharing for Multi-Agent Hybrid DRL for Multiple Multi-Functional RISs-Aided Downlink NOMA Networks](https://mlsanity.com/paper/2601.00538/) (2026)
- [Hybrid TD3: Overestimation Bias Analysis and Stable Policy Optimization for Hybrid Action Space](https://mlsanity.com/paper/2603.01302/) (2026)

## Abstract (arXiv)

Most existing deep reinforcement learning (DRL) frameworks consider either discrete action space or continuous action space solely. Motivated by applications in computer games, we consider the scenario with discrete-continuous hybrid action space. To handle hybrid action space, previous works either approximate the hybrid space by discretization, or relax it into a continuous set. In this paper, we propose a parametrized deep Q-network (P- DQN) framework for the hybrid action space without approximation or relaxation. Our algorithm combines the spirits of both DQN (dealing with discrete action space) and DDPG (dealing with continuous action space) by seamlessly integrating them. Empirical results on a simulation example, scoring a goal in simulated RoboCup soccer and the solo mode in game King of Glory (KOG) validate the efficiency and effectiveness of our method.