mlsanityData to 7 Oct 2026

ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularization

arXiv 2402.14528 ↗First posted 22 Feb 2024cs.LG cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

The varying significance of distinct primitive behaviors during the policy learning process has been overlooked by prior model-free RL algorithms. Leveraging this insight, we explore the causal relationship between different action dimensions and rewards to evaluate the significance of various primitive behaviors during training. We introduce a causality-aware entropy term that effectively identifies and prioritizes actions with high potential impacts for efficient exploration. Furthermore, to prevent excessive focus on specific primitive behaviors, we analyze the gradient dormancy phenomenon and introduce a dormancy-guided reset mechanism to further enhance the efficacy of our method. Our proposed algorithm, ACE: Off-policy Actor-critic with Causality-aware Entropy regularization, demonstrates a substantial performance advantage across 29 diverse continuous control tasks spanning 7 domains compared to model-free RL baselines, which underscores the effectiveness, versatility, and efficient sample efficiency of our approach. Benchmark results and videos are available at https://ace-rl.github.io/.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 54 of its 77 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

54 of 77 references.

References · 54

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2018Exploration by Random Network Distillationcited by 966
  10. 2018DeepMind Control Suitecited by 935
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  12. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
38 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  2. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  3. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  4. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  5. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  6. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  7. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  8. 2017Count-Based Exploration with Neural Density Modelscited by 255
  9. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  10. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  11. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  12. 2019Efficient Exploration via State Marginal Matchingcited by 186
  13. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  14. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  15. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  16. 2019Causal Discovery from Heterogeneous/Nonstationary Data with Independent Changescited by 166
  17. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  18. 2022DAGMA: Learning DAGs via M-matrices and a Log-Determinant Acyclicity Characterizationcited by 135
  19. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  20. 2020Adaptive Reward-Free Explorationcited by 77
  21. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 71
  22. 2019ROBEL: Robotics Benchmarks for Learning with Low-Cost Robotscited by 70
  23. 2021panda-gym: Open-source goal-conditioned environments for robotic learningcited by 56
  24. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 52
  25. 2021AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learningcited by 46
  26. 2021UCB Momentum Q-learning: Correcting the bias without forgettingcited by 45
  27. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
  28. 2023A Survey on Causal Reinforcement Learningcited by 43
  29. 2019Maximum Entropy-Regularized Multi-Goal Reinforcement Learningcited by 39
  30. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  31. 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 33
  32. 2021Action-Sufficient State Representation Learning for Control with Structural Constraintscited by 31
  33. 2022Factored Adaptation for Non-Stationary Reinforcement Learningcited by 30
  34. 2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28
  35. 2022Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversitycited by 22
  36. 2023Causal Reinforcement Learning: A Surveycited by 22
  37. 2020Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Explorationcited by 20
  38. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 6; 2026: 3.

16 papers.

Sort:

By area:

  1. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  2. 2024Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasetscited by 23
  3. 2024MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learningcited by 20
  4. 2025Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learningcited by 17
  5. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  6. 2024Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusioncited by 12
  7. 2025Rethinking Bimanual Robotic Manipulation: Learning with Decoupled Interaction Frameworkcited by 11
  8. 2025Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoningcited by 6
  9. 2026Failure-Aware RL: Reliable Offline-to-Online Reinforcement Learning with Self-Recovery for Real-World Manipulationcited by 6
  10. 2024UVIS: Unsupervised Video Instance Segmentationcited by 5
  11. 2024BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learningcited by 3
  12. 2024Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrationscited by 2
4 more
  1. 2025Causality-informed Anomaly Detection in Partially Observable Sensor Networks: Moving beyond Correlationscited by 2
  2. 2026Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modelingcited by 2
  3. 2025Causal Policy Learning in Reinforcement Learning: Backdoor-Adjusted Soft Actor-Criticcited by 0
  4. 2026SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropycited by 0