mlsanityData to 7 Oct 2026

Action Redundancy in Reinforcement Learning

arXiv 2102.11329 ↗First posted 22 Feb 2021cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Maximum Entropy (MaxEnt) reinforcement learning is a powerful learning paradigm which seeks to maximize return under entropy regularization. However, action entropy does not necessarily coincide with state entropy, e.g., when multiple actions produce the same transition. Instead, we propose to maximize the transition entropy, i.e., the entropy of next states. We show that transition entropy can be described by two terms; namely, model-dependent transition entropy and action redundancy. Particularly, we explore the latter in both deterministic and stochastic settings and develop tractable approximation methods in a near model-free setup. We construct algorithms to minimize action redundancy and demonstrate their effectiveness on a synthetic environment with multiple redundant actions as well as contemporary benchmarks in Atari and Mujoco. Our results suggest that action redundancy is a fundamental problem in reinforcement learning.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 19 of its 37 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

19 of 37 references.

References · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2017A Survey on Multi-Task Learningcited by 838
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  6. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  7. 2016Variational Intrinsic Controlcited by 262
  8. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  9. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  10. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  11. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  12. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
7 more
  1. 2018Learn What Not to Learn: Action Elimination with Deep Reinforcement Learningcited by 77
  2. 2019Learning Action Representations for Reinforcement Learningcited by 75
  3. 2021Geometric Entropic Explorationcited by 32
  4. 2019The Natural Language of Actionscited by 28
  5. 2019An Intrinsically-Motivated Approach for Learning Highly Exploring and Fast Mixing Policiescited by 20
  6. 2019Geometric Losses for Distributional Learningcited by 7
  7. 2019Autonomous exploration for navigating in non-stationary CMPscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 0; 2023: 2; 2024: 1; 2025: 3; 2026: 2.

9 papers.

  1. 2025On Entropy Control in LLM-RL Algorithmscited by 39
  2. 2025InfoBridge: Mutual Information estimation via Bridge Matchingcited by 7
  3. 2023No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learningcited by 6
  4. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
  5. 2021Locality Matters: A Scalable Value Decomposition Approach for Cooperative Multi-Agent Reinforcement Learningcited by 3
  6. 2023DCT: Dual Channel Training of Action Embeddings for Reinforcement Learning with Large Discrete Action Spacescited by 0
  7. 2024Safety through Permissibility: Shield Construction for Fast and Safe Reinforcement Learningcited by 0
  8. 2025Reducing Action Space for Deep Reinforcement Learning via Causal Effect Estimationcited by 0
  9. 2026Not All Transitions Matter: Evidence from PPOcited by 0