mlsanityData to 7 Oct 2026

Near-optimal Policy Identification in Active Reinforcement Learning

arXiv 2212.09510 ↗First posted 19 Dec 2022stat.ML cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Many real-world reinforcement learning tasks require control of complex dynamical systems that involve both costly data acquisition processes and large state spaces. In cases where the transition dynamics can be readily evaluated at specified states (e.g., via a simulator), agents can operate in what is often referred to as planning with a generative model. We propose the AE-LSVI algorithm for best-policy identification, a novel variant of the kernelized least-squares value iteration (LSVI) algorithm that combines optimism with pessimism for active exploration (AE). AE-LSVI provably identifies a near-optimal policy uniformly over an entire state space and achieves polynomial sample complexity guarantees that are independent of the number of states. When specialized to the recently introduced offline contextual Bayesian optimization setting, our algorithm achieves improved sample complexity bounds. Experimentally, we demonstrate that AE-LSVI outperforms other RL algorithms in a variety of environments when robustness to the initial state is required.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 31 of its 64 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

31 of 64 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2016Deep Exploration via Bootstrapped DQNcited by 518
  7. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  8. 2020First return, then explorecited by 256
  9. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  10. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 199
  11. 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 180
  12. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
19 more
  1. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  2. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
  3. 2019Learning with Good Feature Representations in Bandits and in RL with a Generative Modelcited by 124
  4. 2018Model-Based Active Explorationcited by 105
  5. 2014Best-Arm Identification in Linear Banditscited by 98
  6. 2019Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonflycited by 66
  7. 2020Distributionally Robust Bayesian Optimizationcited by 60
  8. 2020Symmetry-Aware Actor-Critic for 3D Molecular Designcited by 55
  9. 2021Misspecified Gaussian Process Bandit Optimizationcited by 52
  10. 2020Reinforcement Learning for Molecular Design Guided by Quantum Mechanicscited by 51
  11. 2021Bayesian Algorithm Execution: Estimating Computable Properties of Black-box Functions Using Mutual Informationcited by 33
  12. 2022Nearly Minimax Optimal Reinforcement Learning with Linear Function Approximationcited by 30
  13. 2020Kernel-Based Reinforcement Learning: A Finite-Time Analysiscited by 23
  14. 2021Efficient Local Planning with Linear Function Approximationcited by 21
  15. 2021Sample-Efficient Reinforcement Learning for Linearly-Parameterized MDPs with a Generative Modelcited by 18
  16. 2020Efficient Planning in Large MDPs with Weak Linear Function Approximationcited by 17
  17. 2021An Experimental Design Perspective on Model-Based Reinforcement Learningcited by 16
  18. 2022Exploration via Planning for Information about the Optimal Trajectorycited by 8
  19. 2022Provably Efficient Kernelized Q-Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 1; 2025: 1; 2026: 1.

7 papers.

  1. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  2. 2023Sample Efficient Preference Alignment in LLMs via Active Explorationcited by 18
  3. 2023Distributionally Robust Model-based Reinforcement Learning with Large State Spacescited by 17
  4. 2023Kernelized Offline Contextual Dueling Banditscited by 4
  5. 2023Multitask Learning with No Regret: from Improved Confidence Bounds to Active Learningcited by 4
  6. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
  7. 2025Offline Clustering of Preference Learning with Active-data Augmentationcited by 0