mlsanityData to 14 Oct 2026

Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions

arXiv 2512.20831 ↗First posted 23 Dec 2025cs.AI cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Real-world sequential decision-making often involves parameterized action spaces that require both, decisions regarding discrete actions and decisions about continuous action parameters governing how an action is executed. Existing approaches exhibit severe limitations in this setting -- planning methods demand hand-crafted action models, and standard reinforcement learning (RL) algorithms are designed for either discrete or continuous actions but not both, and the few RL methods that handle parameterized actions typically rely on domain-specific engineering and fail to exploit the latent structure of these spaces. This paper extends the scope of RL algorithms to long-horizon, sparse-reward settings with parameterized actions by enabling agents to autonomously learn both state and action abstractions online. We introduce algorithms that progressively refine these abstractions during learning, increasing fine-grained detail in the critical regions of the state-action space where greater resolution improves performance. Across several continuous-state, parameterized-action domains, our abstraction-driven approach enables TD(λ) to achieve markedly higher sample efficiency than state-of-the-art baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 29 references that are in the map. First posted on arXiv 23 Dec 2025.

Builds on

20 of 29 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Expediting Reinforcement Learning by Incorporating Knowledge About Temporal Causality in the Environment17 Oct
  2. 2024Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning20 Dec
  3. 2024Continuous Control with Coarse-to-fine Reinforcement Learning10 Jul
  4. 2024Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning23 Jan

Other references · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  5. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  6. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  7. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 191
  8. 2015Deep Reinforcement Learning in Parameterized Action Spacecited by 68
  9. 2015Reinforcement Learning with Parameterized Actionscited by 63
  10. 2018Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Spacecited by 58
  11. 2019Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Spacecited by 54
  12. 2021HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representationcited by 35
4 more
  1. 2023Bridging RL Theory and Practice with the Effective Horizoncited by 31
  2. 2019Multi-Pass Q-Networks for Deep Reinforcement Learning with Parameterised Action Spacescited by 24
  3. 2021Polynomial Time Reinforcement Learning in Factored State MDPs with Linear Value Functionscited by 4
  4. 2024Learning Abstract World Model for Value-preserving Planning with Optionscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimization
  2. 2025Learning with Expert Abstractions for Efficient Multi-Task Continuous Control
  3. 2024Action abstractions for amortized sampling
  4. 2026Knowledge- and Gradient-Guided Reinforcement Learning for Parametrized Action Markov Decision Processes
  5. 2024Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrations
  6. 2024RL-CFR: Improving Action Abstraction for Imperfect Information Extensive-Form Games with Reinforcement Learning
  7. 2024Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control
  8. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey