mlsanityData to 14 Oct 2026

Adaptive Exploration for Data-Efficient General Value Function Evaluations

arXiv 2405.07838 ↗First posted 13 May 2024cs.LG cs.AIcited by 2

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

General Value Functions (GVFs) (Sutton et al., 2011) represent predictive knowledge in reinforcement learning. Each GVF computes the expected return for a given policy, based on a unique reward. Existing methods relying on fixed behavior policies or pre-collected data often face data efficiency issues when learning multiple GVFs in parallel using off-policy methods. To address this, we introduce GVFExplorer, which adaptively learns a single behavior policy that efficiently collects data for evaluating multiple GVFs in parallel. Our method optimizes the behavior policy by minimizing the total variance in return across GVFs, thereby reducing the required environmental interactions. We use an existing temporal-difference-style variance estimator to approximate the return variance. We prove that each behavior policy update decreases the overall mean squared error in GVF predictions. We empirically show our method's performance in tabular and nonlinear function approximation settings, including Mujoco environments, with stationary and non-stationary reward signals, optimizing data usage and reducing prediction errors across multiple GVFs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 16 of its 46 references that are in the map. First posted on arXiv 13 May 2024.

Builds on

16 of 46 references.

References · 16

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2015Prioritized Experience Replaycited by 1,099
  4. 2018Exploration by Random Network Distillationcited by 993
  5. 2018DeepMind Control Suitecited by 956
  6. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  7. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  8. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  9. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  10. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 132
  11. 2017Eigenoption Discovery through the Deep Successor Representationcited by 86
  12. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
4 more
  1. 2017Data-Efficient Policy Evaluation Through Behavior Policy Searchcited by 26
  2. 2018General Value Function Networkscited by 16
  3. 2021Variance Penalized On-Policy and Off-Policy Actor-Criticcited by 10
  4. 2022Continual Auxiliary Task Learningcited by 9

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Adaptive Exploration for Multi-Reward Multi-Policy Evaluationcited by 6
  2. 2025Clustered KL-barycenter design for policy evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DataFlex-RL: An Evaluation Platform for RLVR Data Policies
  2. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Space
  3. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization
  4. 2022A Generalized Bootstrap Target for Value-Learning, Efficiently Combining Value and Feature Predictions
  5. 2026V0.5: Generalist Value Model as a Prior for Sparse RL Rollouts
  6. 2026Le Critique: Privileged Value Functions for LLM Reinforcement Learning
  7. 2025General Exploratory Bonus for Optimistic Exploration in RLHF
  8. 2025Pretrain Value, Not Reward: Decoupled Value Policy Optimization