# Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space

arXiv: https://arxiv.org/abs/1903.01344 · HTML: https://mlsanity.com/paper/1903.01344/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2019.

## Built on by (most cited first, 30 of 54)

- [Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks](https://mlsanity.com/paper/2110.03655/) (2021)
- [Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives](https://mlsanity.com/paper/2110.15360/) (2021)
- [C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters](https://mlsanity.com/paper/2309.11351/) (2023)
- [LLM4EDA: Emerging Progress in Large Language Models for Electronic Design Automation](https://mlsanity.com/paper/2401.12224/) (2023)
- [HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation](https://mlsanity.com/paper/2109.05490/) (2021)
- [Deep hierarchical reinforcement agents for automated penetration testing](https://mlsanity.com/paper/2109.06449/) (2021)
- [Multi-Agent Collaborative Inference via DNN Decoupling: Intermediate Feature Compression and Edge Learning](https://mlsanity.com/paper/2205.11854/) (2022)
- [Multi-user Co-inference with Batch Processing Capable Edge Server](https://mlsanity.com/paper/2206.06304/) (2022)
- [Flowsheet synthesis through hierarchical reinforcement learning and graph neural networks](https://mlsanity.com/paper/2207.12051/) (2022)
- [RELIEF: Reinforcement Learning Empowered Graph Feature Prompt Tuning](https://mlsanity.com/paper/2408.03195/) (2024)
- [Model-based Reinforcement Learning for Parameterized Action Spaces](https://mlsanity.com/paper/2404.03037/) (2024)
- [Meta-Learning Parameterized Skills](https://mlsanity.com/paper/2206.03597/) (2022)
- [Generalising Discrete Action Spaces with Conditional Action Trees](https://mlsanity.com/paper/2104.07294/) (2021)
- [One model Packs Thousands of Items with Recurrent Conditional Query Learning](https://mlsanity.com/paper/2111.06726/) (2021)
- [Reinforcement learning for traffic signal control in hybrid action space](https://mlsanity.com/paper/2211.12956/) (2022)
- [Risk-anticipatory autonomous driving strategies considering vehicles' weights, based on hierarchical deep reinforcement learning](https://mlsanity.com/paper/2401.08661/) (2023)
- [Exploiting Hybrid Policy in Reinforcement Learning for Interpretable Temporal Logic Manipulation](https://mlsanity.com/paper/2412.20338/) (2024)
- [A semi-centralized multi-agent RL framework for efficient irrigation scheduling](https://mlsanity.com/paper/2408.08442/) (2024)
- [Optimizing Navigation And Chemical Application in Precision Agriculture With Deep Reinforcement Learning And Conditional Action Tree](https://mlsanity.com/paper/2503.17985/) (2025)
- [TRIDENT: Breaking the Hybrid-Safety-Physics Coupling for Provably Safe Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2606.18308/) (2026)
- [A Survey of Learning on Small Data: Generalization, Optimization, and Challenge](https://mlsanity.com/paper/2207.14443/) (2022)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [Adaptive Frequency Green Light Optimal Speed Advisory based on Hybrid Actor-Critic Reinforcement Learning](https://mlsanity.com/paper/2306.04660/) (2023)
- [Integrating Reinforcement Learning and Model Predictive Control with Applications to Microgrids](https://mlsanity.com/paper/2409.11267/) (2024)
- [AOAD-MAT: Transformer-based multi-agent deep reinforcement learning model considering agents' order of action decisions](https://mlsanity.com/paper/2510.13343/) (2025)
- [Logic-informed reinforcement learning for cross-domain optimization of large-scale cyber-physical systems](https://mlsanity.com/paper/2511.00806/) (2025)
- [Large-scale automatic carbon ion treatment planning for head and neck cancers via parallel multi-agent reinforcement learning](https://mlsanity.com/paper/2511.02314/) (2025)
- [Dmsh: A Multi-Agent Reinforcement Learning Framework for All-Quad Mesh Generation](https://mlsanity.com/paper/2606.10601/) (2026)
- [SAGE: Generating Symbolic Goals for Myopic Models in Deep Reinforcement Learning](https://mlsanity.com/paper/2203.05079/) (2022)
- [Mixed-Integer Optimal Control via Reinforcement Learning: A Case Study on Hybrid Electric Vehicle Energy Management](https://mlsanity.com/paper/2305.01461/) (2023)

## Abstract (arXiv)

In this paper we propose a hybrid architecture of actor-critic algorithms for reinforcement learning in parameterized action space, which consists of multiple parallel sub-actor networks to decompose the structured action space into simpler action spaces along with a critic network to guide the training of all sub-actor networks. While this paper is mainly focused on parameterized action space, the proposed architecture, which we call hybrid actor-critic, can be extended for more general action spaces which has a hierarchical structure. We present an instance of the hybrid actor-critic architecture based on proximal policy optimization (PPO), which we refer to as hybrid proximal policy optimization (H-PPO). Our experiments test H-PPO on a collection of tasks with parameterized action space, where H-PPO demonstrates superior performance over previous methods of parameterized action reinforcement learning.