Data to 7 Oct 2026

Efficient Exploration via State Marginal Matching

arXiv 1906.05274 ↗First posted 12 Jun 2019cs.LG cs.AI cs.ROcited by 186

Not on the map. Why?

Cited by 29 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Exploration is critical to a reinforcement learning agent's performance in its given environment. Prior exploration methods are often based on using heuristic auxiliary predictions to guide policy behavior, lacking a mathematically-grounded objective with clear properties. In contrast, we recast exploration as a problem of State Marginal Matching (SMM), where we aim to learn a policy for which the state marginal distribution matches a given target state distribution. The target distribution is a uniform distribution in most cases, but can incorporate prior knowledge if available. In effect, SMM amortizes the cost of learning to explore in a given environment. The SMM objective can be viewed as a two-player, zero-sum game between a state density model and a parametric policy, an idea that we use to build an algorithm for optimizing the SMM objective. Using this formalism, we further demonstrate that prior work approximately maximizes the SMM objective, offering an explanation for the success of these methods. On both simulated and real-world tasks, we demonstrate that agents that directly optimize the SMM objective explore faster and adapt more quickly to new tasks as compared to prior exploration methods.

Key facts. Cited by 186 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 2; 2020: 1; 2021: 42; 2022: 35; 2023: 41; 2024: 29; 2025: 20; 2026: 16.

186 papers.

Sort:

By area:

2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
2021Reinforcement Learning with Prototypical Representationscited by 179
2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
2021APS: Active Pretraining with Successor Featurescited by 109
2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93

38 more

2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
2021Reward is enough for convex MDPscited by 82
2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
2023XSkill: Cross Embodiment Skill Discoverycited by 74
2019State Alignment-based Imitation Learningcited by 73
2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
2022Lipschitz-constrained Unsupervised Skill Discoverycited by 60
2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 54
2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
2023Controllability-Aware Unsupervised Skill Discoverycited by 49
2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 46
2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
2021The Information Geometry of Unsupervised Reinforcement Learningcited by 36
2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
2021Geometric Entropic Explorationcited by 32
2023Fast Rates for Maximum Entropy Explorationcited by 32
2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 29
2023Behavior Contrastive Learning for Unsupervised Skill Discoverycited by 26
2023Future-conditioned Unsupervised Pretraining for Decision Transformercited by 26
2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 25
2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24
2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
2022Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversitycited by 22
2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
2021Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learningcited by 21
2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 21
2023Accelerating Exploration with Unlabeled Prior Datacited by 21

50 most cited of 186 All 186 (JSON)