mlsanityData to 14 Oct 2026

Geometric Active Exploration in Markov Decision Processes: the Benefit of Abstraction

arXiv 2407.13364 ↗First posted 18 Jul 2024cs.LGcited by 6

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

How can a scientist use a Reinforcement Learning (RL) algorithm to design experiments over a dynamical system's state space? In the case of finite and Markovian systems, an area called Active Exploration (AE) relaxes the optimization problem of experiments design into Convex RL, a generalization of RL admitting a wider notion of reward. Unfortunately, this framework is currently not scalable and the potential of AE is hindered by the vastness of experiment spaces typical of scientific discovery applications. However, these spaces are often endowed with natural geometries, e.g., permutation invariance in molecular design, that an agent could leverage to improve the statistical and computational efficiency of AE. To achieve this, we bridge AE and MDP homomorphisms, which offer a way to exploit known geometric structures via abstraction. Towards this goal, we make two fundamental contributions: we extend MDP homomorphisms formalism to Convex RL, and we present, to the best of our knowledge, the first analysis that formally captures the benefit of abstraction via homomorphisms on sample efficiency. Ultimately, we propose the Geometric Active Exploration (GAE) algorithm, which we analyse theoretically and experimentally in environments motivated by problems in scientific discovery.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 50 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

19 of 50 references.

References · 19

  1. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  2. 2020MDP Homomorphic Networks: Group Symmetries in Reinforcement Learningcited by 142
  3. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
  4. 2021Reward is enough for convex MDPscited by 84
  5. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 83
  6. 2020Plannable Approximations to MDP Homomorphisms: Equivariance under Actionscited by 61
  7. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 56
  8. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  9. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 30
  10. 2019Active Exploration in Markov Decision Processescited by 27
  11. 2022Active Exploration for Inverse Reinforcement Learningcited by 27
  12. 2022Continuous MDP Homomorphisms and Homomorphic Policy Gradientcited by 27
7 more
  1. 2021Multi-Agent MDP Homomorphic Networkscited by 25
  2. 2020Active Model Estimation in Markov Decision Processescited by 20
  3. 2022Active Exploration via Experiment Design in Markov Chainscited by 19
  4. 2020Curiosity in exploring chemical space: Intrinsic rewards for deep molecular reinforcement learningcited by 16
  5. 2017Symmetry Learning for Function Approximation in Reinforcement Learningcited by 14
  6. 2018Online Abstraction with MDP Homomorphisms for Deep Learningcited by 13
  7. 2021Expert-Guided Symmetry Detection in Markov Decision Processescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuningcited by 17
  2. 2025Provable Maximum Entropy Manifold Exploration via Diffusion Modelscited by 14
  3. 2026How to Interpret Agent Behaviorcited by 10
  4. 2026Efficient Tail-Aware Generative Optimization via Flow Model Fine-Tuningcited by 8
  5. 2026Verifier-Constrained Flow Expansion for Discovery Beyond the Datacited by 7
  6. 2026A Unified Density Operator View of Flow Control and Mergingcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Model-Free Active Exploration in Reinforcement Learning
  2. 2025An Optimal Policy for Learning Controllable Dynamics by Exploration
  3. 2026Mathematical methods of reinforcement learning
  4. 2026Sample Complexity of Equivariant Reinforcement Learning
  5. 2026Active Inference as a Convex Markov Decision Process
  6. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design
  7. 2026Policy Gradient Methods for Non-Markovian Reinforcement Learning
  8. 2026Topology-Aware State Abstraction with Tangle Cores for Markov Decision Processes