mlsanityData to 7 Oct 2026

Reducing Action Space for Deep Reinforcement Learning via Causal Effect Estimation

arXiv 2501.14543 ↗First posted 24 Jan 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Intelligent decision-making within large and redundant action spaces remains challenging in deep reinforcement learning. Considering similar but ineffective actions at each step can lead to repetitive and unproductive trials. Existing methods attempt to improve agent exploration by reducing or penalizing redundant actions, yet they fail to provide quantitative and reliable evidence to determine redundancy. In this paper, we propose a method to improve exploration efficiency by estimating the causal effects of actions. Unlike prior methods, our approach offers quantitative results regarding the causality of actions for one-step transitions. We first pre-train an inverse dynamics model to serve as prior knowledge of the environment. Subsequently, we classify actions across the entire action space at each time step and estimate the causal effect of each action to suppress redundant actions during exploration. We provide a theoretical analysis to demonstrate the effectiveness of our method and present empirical results from simulations in environments with redundant actions to evaluate its performance. Our implementation is available at https://github.com/agi-brain/cee.git.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 24 Jan 2025.

Builds on

22 of 27 references.

References · 22

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  4. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  5. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  6. 2019Causality for Machine Learningcited by 242
  7. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  8. 2020A Closer Look at Invalid Action Masking in Policy Gradient Algorithmscited by 160
  9. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 158
  10. 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
  11. 2018Learn What Not to Learn: Action Elimination with Deep Reinforcement Learningcited by 77
  12. 2019Learning Action Representations for Reinforcement Learningcited by 75
10 more
  1. 2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 70
  2. 2020Action Space Shaping in Deep Reinforcement Learningcited by 45
  3. 2021Gym-μRTS: Toward Affordable Full Game Real-time Strategy Games Research with Deep Reinforcement Learningcited by 19
  4. 2023Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approachcited by 17
  5. 2020Disentangling causal effects for hierarchical reinforcement learningcited by 9
  6. 2021Action Redundancy in Reinforcement Learningcited by 9
  7. 2020Reinforcement Learning of Causal Variables Using Mediation Analysiscited by 6
  8. 2023No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learningcited by 6
  9. 2024Variable-Agnostic Causal Exploration for Reinforcement Learningcited by 4
  10. 2024Efficient Reinforcement Learning of Task Planners for Robotic Palletization through Iterative Action Masking Learningcited by 2

Built on by

Not cited yet.