mlsanityData to 7 Oct 2026

Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration

arXiv 2305.18258 ↗First posted 29 May 2023cs.LG cs.AI cs.GTcited by 20

Not on the map. Why?

Abstract (arXiv)

In online reinforcement learning (online RL), balancing exploration and exploitation is crucial for finding an optimal policy in a sample-efficient way. To achieve this, existing sample-efficient online RL algorithms typically consist of three components: estimation, planning, and exploration. However, in order to cope with general function approximators, most of them involve impractical algorithmic components to incentivize exploration, such as optimization within data-dependent level-sets or complicated sampling procedures. To address this challenge, we propose an easy-to-implement RL framework called textitMaximize to Explore (textttMEX), which only needs to optimize unconstrainedly a single objective that integrates the estimation and planning components while balancing exploration and exploitation automatically. Theoretically, we prove that textttMEX achieves a sublinear regret with general function approximations for Markov decision processes (MDP) and is further extendable to two-player zero-sum Markov games (MG). Meanwhile, we adapt deep RL baselines to design practical versions of textttMEX, in both model-free and model-based manners, which can outperform baselines by a stable margin in various MuJoCo environments with sparse rewards. Compared with existing sample-efficient online RL algorithms with general function approximations, textttMEX achieves similar sample efficiency while enjoying a lower computational cost and is more compatible with modern deep RL methods.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 59 of its 93 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

59 of 93 references.

References · 59

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  6. 2018Exploration by Random Network Distillationcited by 966
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  9. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  10. 2016Deep Exploration via Bootstrapped DQNcited by 518
  11. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  12. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
38 more
  1. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  2. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  3. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  4. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  5. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  6. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  7. 2021The Statistical Complexity of Interactive Decision Makingcited by 208
  8. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  9. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 199
  10. 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 180
  11. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  12. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  13. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  14. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
  15. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  16. 2020Provable Self-Play Algorithms for Competitive Reinforcement Learningcited by 112
  17. 2019Optimism in Reinforcement Learning with Generalized Linear Function Approximationcited by 111
  18. 2020Near-Optimal Reinforcement Learning with Self-Playcited by 108
  19. 2020A Sharp Analysis of Model-based Reinforcement Learning with Self-Playcited by 104
  20. 2019Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensemblescited by 100
  21. 2019Frequentist Regret Bounds for Randomized Least-Squares Value Iterationcited by 100
  22. 2020Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibriumcited by 100
  23. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  24. 2022The Role of Coverage in Online Reinforcement Learningcited by 82
  25. 2019A survey on intrinsic motivation in reinforcement learningcited by 79
  26. 2020Sample-Efficient Reinforcement Learning of Undercomplete POMDPscited by 66
  27. 2021Feel-Good Thompson Sampling for Contextual Bandits and Reinforcement Learningcited by 65
  28. 2017Ensemble Samplingcited by 62
  29. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  30. 2021The Power of Exploiter: Provable Multi-Agent RL in Large State Spacescited by 48
  31. 2021Towards General Function Approximation in Zero-Sum Markov Gamescited by 42
  32. 2022PAC Reinforcement Learning for Predictive State Representationscited by 42
  33. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  34. 2022Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systemscited by 35
  35. 2022Model-based RL with Optimistic Posterior Sampling: Structural Conditions and Sample Complexitycited by 31
  36. 2022A General Framework for Sample-Efficient Function Approximation in Reinforcement Learningcited by 31
  37. 2022A Provably Efficient Model-Free Posterior Sampling Method for Episodic Reinforcement Learningcited by 30
  38. 2023Tight Guarantees for Interactive Decision Making with the Decision-Estimation Coefficientcited by 29

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 12; 2025: 4; 2026: 2.

20 papers.

Sort:

By area:

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  3. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  4. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  5. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  6. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  7. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  8. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  9. 2024Combinatorial Multivariant Multi-Armed Bandits with Applications to Episodic Reinforcement Learning and Beyondcited by 7
  10. 2025Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limitscited by 7
  11. 2024Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgentcited by 6
  12. 2024Provably and Practically Efficient Adversarial Imitation Learning with General Function Approximationcited by 6
8 more
  1. 2023Horizon-Free and Instance-Dependent Regret Bounds for Reinforcement Learning with General Function Approximationcited by 4
  2. 2024Provably Efficient Information-Directed Sampling Algorithms for Multi-Agent Reinforcement Learningcited by 2
  3. 2026Learning Approximate Nash Equilibria in Cooperative Multi-Agent Reinforcement Learning via Mean-Field Subsamplingcited by 2
  4. 2024Provably Efficient Exploration in Inverse Constrained Reinforcement Learningcited by 1
  5. 2025Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Gamescited by 1
  6. 2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignmentcited by 1
  7. 2026Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithmscited by 1
  8. 2025MAGIC-MASK: Multi-Agent Guided Inter-Agent Collaboration with Mask-Based Explainability for Reinforcement Learningcited by 0