mlsanityData to 7 Oct 2026

SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimation

arXiv 2512.10042 ↗First posted 10 Dec 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

In the unsupervised pre-training for reinforcement learning, the agent aims to learn a prior policy for downstream tasks without relying on task-specific reward functions. We focus on state entropy maximization (SEM), where the goal is to learn a policy that maximizes the entropy of the state stationary distribution. In this paper, we introduce SEMDICE, a principled off-policy algorithm that computes an SEM policy from an arbitrary off-policy dataset, which optimizes the policy directly within the space of stationary distributions. SEMDICE computes a single, stationary Markov state-entropy-maximizing policy from an arbitrary off-policy dataset. Experimental results demonstrate that SEMDICE outperforms baseline algorithms in maximizing state entropy while achieving the best adaptation efficiency for downstream tasks among SEM-based unsupervised RL pre-training methods.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 10 Dec 2025.

Builds on

39 of 52 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2015Continuous control with deep reinforcement learningcited by 3,279
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2021Convex optimizationcited by 2,479
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  10. 2018Exploration by Random Network Distillationcited by 966
  11. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  12. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
27 more
  1. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  2. 2019A Theory of Regularized Markov Decision Processescited by 279
  3. 2019Self-Supervised Exploration via Disagreementcited by 263
  4. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  5. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  6. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 209
  7. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  8. 2019Efficient Exploration via State Marginal Matchingcited by 186
  9. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  10. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  11. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  12. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  13. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  14. 2021APS: Active Pretraining with Successor Featurescited by 109
  15. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  16. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  17. 2020Adaptive Reward-Free Explorationcited by 77
  18. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  19. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  20. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 36
  21. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  22. 2023Fast Rates for Maximum Entropy Explorationcited by 32
  23. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  24. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
  25. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  26. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  27. 2020Active Model Estimation in Markov Decision Processescited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Bellman Meets Lyapunov: Unsupervised Reinforcement Learning via Mastering Chaoscited by 0