# Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces

arXiv: https://arxiv.org/abs/1903.04959 · HTML: https://mlsanity.com/paper/1903.04959/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Mar 2019.

## Built on by (most cited first, 16 of 16)

- [Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives](https://mlsanity.com/paper/2110.15360/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation](https://mlsanity.com/paper/2109.05490/) (2021)
- [Survey of Recent Multi-Agent Reinforcement Learning Algorithms Utilizing Centralized Training](https://mlsanity.com/paper/2107.14316/) (2021)
- [Model-based Reinforcement Learning for Parameterized Action Spaces](https://mlsanity.com/paper/2404.03037/) (2024)
- [Meta-Learning Parameterized Skills](https://mlsanity.com/paper/2206.03597/) (2022)
- [Subequivariant Reinforcement Learning Framework for Coordinated Motion Control](https://mlsanity.com/paper/2403.15100/) (2024)
- [TRIDENT: Breaking the Hybrid-Safety-Physics Coupling for Provably Safe Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2606.18308/) (2026)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [An Efficient Multi-Robot Arm Coordination Strategy for Pick-and-Place Tasks using Reinforcement Learning](https://mlsanity.com/paper/2409.13511/) (2024)
- [UAV aided Metaverse over Wireless Communications: A Reinforcement Learning Approach](https://mlsanity.com/paper/2301.01474/) (2023)
- [Learn A Flexible Exploration Model for Parameterized Action Markov Decision Processes](https://mlsanity.com/paper/2501.02774/) (2025)
- [Distributions as Actions: A Unified Framework for Diverse Action Spaces](https://mlsanity.com/paper/2506.16608/) (2025)
- [A Hybrid Proactive And Predictive Framework For Edge Cloud Resource Management](https://mlsanity.com/paper/2511.16075/) (2025)
- [CHDP: Cooperative Hybrid Diffusion Policies for Reinforcement Learning in Parameterized Action Space](https://mlsanity.com/paper/2601.05675/) (2026)
- [ASALT: Adaptive State Alignment for Lateral Transfer in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2606.24601/) (2026)

## Abstract (arXiv)

Deep Reinforcement Learning (DRL) has been applied to address a variety of cooperative multi-agent problems with either discrete action spaces or continuous action spaces. However, to the best of our knowledge, no previous work has ever succeeded in applying DRL to multi-agent problems with discrete-continuous hybrid (or parameterized) action spaces which is very common in practice. Our work fills this gap by proposing two novel algorithms: Deep Multi-Agent Parameterized Q-Networks (Deep MAPQN) and Deep Multi-Agent Hierarchical Hybrid Q-Networks (Deep MAHHQN). We follow the centralized training but decentralized execution paradigm: different levels of communication between different agents are used to facilitate the training process, while each agent executes its policy independently based on local observations during execution. Our empirical results on several challenging tasks (simulated RoboCup Soccer and game Ghost Story) show that both Deep MAPQN and Deep MAHHQN are effective and significantly outperform existing independent deep parameterized Q-learning method.