# Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration

arXiv: https://arxiv.org/abs/2006.01419 · HTML: https://mlsanity.com/paper/2006.01419/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2020.

## Built on by (most cited first, 20 of 20)

- [Adversarially Guided Actor-Critic](https://mlsanity.com/paper/2102.04376/) (2021)
- [A Max-Min Entropy Framework for Reinforcement Learning](https://mlsanity.com/paper/2106.10517/) (2021)
- [Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic](https://mlsanity.com/paper/2306.02865/) (2023)
- [ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularization](https://mlsanity.com/paper/2402.14528/) (2024)
- [Probabilistic Inference in Reinforcement Learning Done Right](https://mlsanity.com/paper/2311.13294/) (2023)
- [AEM: Attention Entropy Maximization for Multiple Instance Learning based Whole Slide Image Classification](https://mlsanity.com/paper/2406.15303/) (2024)
- [Exclusively Penalized Q-learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2405.14082/) (2024)
- [LESSON: Learning to Integrate Exploration Strategies for Reinforcement Learning via an Option Framework](https://mlsanity.com/paper/2310.03342/) (2023)
- [Self-Improving Skill Learning for Robust Skill-based Meta-Reinforcement Learning](https://mlsanity.com/paper/2502.03752/) (2025)
- [PAC-Bayesian Soft Actor-Critic Learning](https://mlsanity.com/paper/2301.12776/) (2023)
- [Learning Diverse Policies with Soft Self-Generated Guidance](https://mlsanity.com/paper/2402.04539/) (2024)
- [Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learning](https://mlsanity.com/paper/2409.19231/) (2024)
- [Off-policy Reinforcement Learning with Model-based Exploration Augmentation](https://mlsanity.com/paper/2510.25529/) (2025)
- [Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularization](https://mlsanity.com/paper/2511.01374/) (2025)
- [Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2602.17062/) (2026)
- [Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2506.21039/) (2025)
- [Entropy Augmented Reinforcement Learning](https://mlsanity.com/paper/2208.09322/) (2022)
- [Enhance Exploration in Safe Reinforcement Learning with Contrastive Representation Learning](https://mlsanity.com/paper/2503.10318/) (2025)
- [HAEPO: History-Aggregated Exploratory Policy Optimization](https://mlsanity.com/paper/2508.18884/) (2025)
- [Complexity-Regularized Proximal Policy Optimization](https://mlsanity.com/paper/2509.20509/) (2025)

## Abstract (arXiv)

In this paper, sample-aware policy entropy regularization is proposed to enhance the conventional policy entropy regularization for better exploration. Exploiting the sample distribution obtainable from the replay buffer, the proposed sample-aware entropy regularization maximizes the entropy of the weighted sum of the policy action distribution and the sample action distribution from the replay buffer for sample-efficient exploration. A practical algorithm named diversity actor-critic (DAC) is developed by applying policy iteration to the objective function with the proposed sample-aware entropy regularization. Numerical results show that DAC significantly outperforms existing recent algorithms for reinforcement learning.