# Reinforcement Learning with Parameterized Actions

arXiv: https://arxiv.org/abs/1509.01644 · HTML: https://mlsanity.com/paper/1509.01644/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2015.

## Built on by (most cited first, 30 of 63)

- [Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models](https://mlsanity.com/paper/2401.03360/) (2023)
- [Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks](https://mlsanity.com/paper/2110.03655/) (2021)
- [Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives](https://mlsanity.com/paper/2110.15360/) (2021)
- [Deep Imitation Learning for Bimanual Robotic Manipulation](https://mlsanity.com/paper/2010.05134/) (2020)
- [VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning](https://mlsanity.com/paper/2410.23156/) (2024)
- [Hierarchical Skills for Efficient Exploration](https://mlsanity.com/paper/2110.10809/) (2021)
- [HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation](https://mlsanity.com/paper/2109.05490/) (2021)
- [Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods](https://mlsanity.com/paper/2102.13045/) (2021)
- [ReLAX: Reinforcement Learning Agent eXplainer for Arbitrary Predictive Models](https://mlsanity.com/paper/2110.11960/) (2021)
- [Practice Makes Perfect: Planning to Learn Skill Parameter Policies](https://mlsanity.com/paper/2402.15025/) (2024)
- [Deep hierarchical reinforcement agents for automated penetration testing](https://mlsanity.com/paper/2109.06449/) (2021)
- [STAP: Sequencing Task-Agnostic Policies](https://mlsanity.com/paper/2210.12250/) (2022)
- [Efficient XAI Techniques: A Taxonomic Survey](https://mlsanity.com/paper/2302.03225/) (2023)
- [RELIEF: Reinforcement Learning Empowered Graph Feature Prompt Tuning](https://mlsanity.com/paper/2408.03195/) (2024)
- [ExoPredicator: Learning Abstract Models of Dynamic Worlds for Robot Planning](https://mlsanity.com/paper/2509.26255/) (2025)
- [PRIME: Scaffolding Manipulation Tasks with Behavior Primitives for Data-Efficient Imitation Learning](https://mlsanity.com/paper/2403.00929/) (2024)
- [Model-based Reinforcement Learning for Parameterized Action Spaces](https://mlsanity.com/paper/2404.03037/) (2024)
- [Meta-Learning Parameterized Skills](https://mlsanity.com/paper/2206.03597/) (2022)
- [Parameterized Decision-making with Multi-modal Perception for Autonomous Driving](https://mlsanity.com/paper/2312.11935/) (2023)
- [Generalising Discrete Action Spaces with Conditional Action Trees](https://mlsanity.com/paper/2104.07294/) (2021)
- [One model Packs Thousands of Items with Recurrent Conditional Query Learning](https://mlsanity.com/paper/2111.06726/) (2021)
- [TMM-Fast: A Transfer Matrix Computation Package for Multilayer Thin-Film Optimization](https://mlsanity.com/paper/2111.13667/) (2021)
- [Reinforcement learning for traffic signal control in hybrid action space](https://mlsanity.com/paper/2211.12956/) (2022)
- [Design Strategy Network: A deep hierarchical framework to represent generative design strategies in complex action spaces](https://mlsanity.com/paper/2110.03760/) (2021)
- [Generalizing Object-Centric Task-Axes Controllers using Keypoints](https://mlsanity.com/paper/2103.10524/) (2021)
- [A reinforcement learning approach to resource allocation in genomic selection](https://mlsanity.com/paper/2107.10901/) (2021)
- [Active Learning of Quantum System Hamiltonians yields Query Advantage](https://mlsanity.com/paper/2112.14553/) (2021)
- [Active Task Randomization: Learning Robust Skills via Unsupervised Generation of Diverse and Feasible Tasks](https://mlsanity.com/paper/2211.06134/) (2022)
- [Exploiting Hybrid Policy in Reinforcement Learning for Interpretable Temporal Logic Manipulation](https://mlsanity.com/paper/2412.20338/) (2024)
- [Impedance Primitive-augmented Hierarchical Reinforcement Learning for Sequential Tasks](https://mlsanity.com/paper/2508.19607/) (2025)

## Abstract (arXiv)

We introduce a model-free algorithm for learning in Markov decision processes with parameterized actions-discrete actions with continuous parameters. At each step the agent must select both which action to use and which parameters to use with that action. We introduce the Q-PAMDP algorithm for learning in these domains, show that it converges to a local optimum, and compare it to direct policy search in the goal-scoring and Platform domains.