mlsanityData to 7 Oct 2026

Learn A Flexible Exploration Model for Parameterized Action Markov Decision Processes

arXiv 2501.02774 ↗First posted 6 Jan 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Hybrid action models are widely considered an effective approach to reinforcement learning (RL) modeling. The current mainstream method is to train agents under Parameterized Action Markov Decision Processes (PAMDPs), which performs well in specific environments. Unfortunately, these models either exhibit drastic low learning efficiency in complex PAMDPs or lose crucial information in the conversion between raw space and latent space. To enhance the learning efficiency and asymptotic performance of the agent, we propose a model-based RL (MBRL) algorithm, FLEXplore. FLEXplore learns a parameterized-action-conditioned dynamics model and employs a modified Model Predictive Path Integral control. Unlike conventional MBRL algorithms, we carefully design the dynamics loss function and reward smoothing process to learn a loose yet flexible model. Additionally, we use the variational lower bound to maximize the mutual information between the state and the hybrid action, enhancing the exploration effectiveness of the agent. We theoretically demonstrate that FLEXplore can reduce the regret of the rollout trajectory through the Wasserstein Metric under given Lipschitz conditions. Our empirical results on several standard benchmarks show that FLEXplore has outstanding learning efficiency and asymptotic performance compared to other baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 56 references that are in the map. First posted on arXiv 6 Jan 2025.

Builds on

39 of 56 references.

References · 39

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  7. 2020Mastering Atari with Discrete World Modelscited by 990
  8. 2019Introduction to Online Convex Optimizationcited by 793
  9. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  10. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  11. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 460
  12. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
27 more
  1. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  2. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  3. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
  4. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 90
  5. 2015Deep Reinforcement Learning in Parameterized Action Spacecited by 65
  6. 2015Reinforcement Learning with Parameterized Actionscited by 61
  7. 2018Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Spacecited by 54
  8. 2019Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Spacecited by 53
  9. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  10. 2021HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representationcited by 33
  11. 2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
  12. 2022Continuous MDP Homomorphisms and Homomorphic Policy Gradientcited by 24
  13. 2019Multi-Pass Q-Networks for Deep Reinforcement Learning with Parameterised Action Spacescited by 22
  14. 2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
  15. 2024Privileged Sensing Scaffolds Reinforcement Learningcited by 18
  16. 2022The Benefits of Model-Based Generalization in Reinforcement Learningcited by 17
  17. 2023Plan To Predict: Learning an Uncertainty-Foreseeing Model for Model-Based Reinforcement Learningcited by 16
  18. 2019Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spacescited by 15
  19. 2021Learning Insertion Primitives with Discrete-Continuous Hybrid Action Space for Robotic Assembly Taskscited by 15
  20. 2023MoVie: Visual Model-Based Policy Adaptation for View Generalizationcited by 15
  21. 2023The Virtues of Laziness in Model-based RL: A Unified Objective and Algorithmscited by 12
  22. 2023DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothingcited by 8
  23. 2020Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learningcited by 7
  24. 2024Model-based Reinforcement Learning for Parameterized Action Spacescited by 7
  25. 2024Dr. Strategy: Model-Based Generalist Agents with Strategic Dreamingcited by 6
  26. 2023Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learningcited by 5
  27. 2024Walk Wisely on Graph: Knowledge Graph Reasoning with Dual Agents via Efficient Guidance-Explorationcited by 2

Built on by

Not cited yet.