# Multi-Pass Q-Networks for Deep Reinforcement Learning with Parameterised Action Spaces

arXiv: https://arxiv.org/abs/1905.04388 · HTML: https://mlsanity.com/paper/1905.04388/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2019.

## Built on by (most cited first, 24 of 24)

- [HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation](https://mlsanity.com/paper/2109.05490/) (2021)
- [Model-based Reinforcement Learning for Parameterized Action Spaces](https://mlsanity.com/paper/2404.03037/) (2024)
- [Meta-Learning Parameterized Skills](https://mlsanity.com/paper/2206.03597/) (2022)
- [Discretionary Lane-Change Decision and Control via Parameterized Soft Actor-Critic for Hybrid Action Space](https://mlsanity.com/paper/2402.15790/) (2024)
- [Reinforcement learning for traffic signal control in hybrid action space](https://mlsanity.com/paper/2211.12956/) (2022)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [Detecting Continuous Integration Skip : A Reinforcement Learning-based Approach](https://mlsanity.com/paper/2405.09657/) (2024)
- [Comparative Analysis of Parameterized Action Actor-Critic Reinforcement Learning Algorithms for Web Search Match Plan Generation](https://mlsanity.com/paper/2510.03064/) (2025)
- [Learning Parameterized Skills from Demonstrations](https://mlsanity.com/paper/2510.24095/) (2025)
- [Deep Multi-Agent Reinforcement Learning with Hybrid Action Spaces based on Maximum Entropy](https://mlsanity.com/paper/2206.05108/) (2022)
- [A further exploration of deep Multi-Agent Reinforcement Learning with Hybrid Action Space](https://mlsanity.com/paper/2208.14447/) (2022)
- [Computational Discovery of Energy-Efficient Heat Treatment for Microstructure Design using Deep Reinforcement Learning](https://mlsanity.com/paper/2209.11259/) (2022)
- [Mixed-Integer Optimal Control via Reinforcement Learning: A Case Study on Hybrid Electric Vehicle Energy Management](https://mlsanity.com/paper/2305.01461/) (2023)
- [Learn A Flexible Exploration Model for Parameterized Action Markov Decision Processes](https://mlsanity.com/paper/2501.02774/) (2025)
- [A Parallel Hybrid Action Space Reinforcement Learning Model for Real-world Adaptive Traffic Signal Control](https://mlsanity.com/paper/2503.14250/) (2025)
- [Multi-Agent Deep Reinforcement Learning for Safe Autonomous Driving with RICS-Assisted MEC](https://mlsanity.com/paper/2503.19418/) (2025)
- [Strategic Communication under Threat: Learning Information Trade-offs in Pursuit-Evasion Games](https://mlsanity.com/paper/2510.07813/) (2025)
- [Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions](https://mlsanity.com/paper/2512.20831/) (2025)
- [Breaking the Grid: Distance-Guided Reinforcement Learning in Large Discrete Action Spaces](https://mlsanity.com/paper/2602.08616/) (2026)
- [Learning Hybrid-Control Policies for High-Precision In-Contact Manipulation Under Uncertainty](https://mlsanity.com/paper/2604.19677/) (2026)
- [Knowledge- and Gradient-Guided Reinforcement Learning for Parametrized Action Markov Decision Processes](https://mlsanity.com/paper/2607.12924/) (2026)
- [Comparative Study of Multi-Agent Actor-Critic Algorithms in Parameterized Action Reinforcement Learning](https://mlsanity.com/paper/2607.19117/) (2026)
- [Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition](https://mlsanity.com/paper/2608.09762/) (2026)
- [Reinforcement Learning to Accelerate Primal-Dual Hybrid Gradient for Linear Programming](https://mlsanity.com/paper/2610.01546/) (2026)

## Abstract (arXiv)

Parameterised actions in reinforcement learning are composed of discrete actions with continuous action-parameters. This provides a framework for solving complex domains that require combining high-level actions with flexible control. The recent P-DQN algorithm extends deep Q-networks to learn over such action spaces. However, it treats all action-parameters as a single joint input to the Q-network, invalidating its theoretical foundations. We analyse the issues with this approach and propose a novel method, multi-pass deep Q-networks, or MP-DQN, to address them. We empirically demonstrate that MP-DQN significantly outperforms P-DQN and other previous algorithms in terms of data efficiency and converged policy performance on the Platform, Robot Soccer Goal, and Half Field Offense domains.