mlsanityData to 14 Oct 2026

Parameter-Based Value Functions

arXiv 2006.09226 ↗First posted 16 Jun 2020cs.LG cs.AI stat.MLcited by 24

Not on the map. Why?

Abstract (arXiv)

Traditional off-policy actor-critic Reinforcement Learning (RL) algorithms learn value functions of a single target policy. However, when value functions are updated to track the learned policy, they forget potentially useful information about old policies. We introduce a class of value functions called Parameter-Based Value Functions (PBVFs) whose inputs include the policy parameters. They can generalize across different policies. PBVFs can evaluate the performance of any policy given a state, a state-action pair, or a distribution over the RL agent's initial states. First we show how PBVFs yield novel off-policy policy gradient theorems. Then we derive off-policy actor-critic algorithms based on PBVFs trained by Monte Carlo or Temporal Difference methods. We show how learned PBVFs can zero-shot learn new policies that outperform any policy seen during training. Finally our algorithms are evaluated on a selection of discrete and continuous control tasks using shallow policies and deep neural networks. Their performance is comparable to state-of-the-art methods.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 8; 2023: 1; 2024: 4; 2025: 6; 2026: 1.

24 papers.

Sort:

By area:

  1. 2021Universal Off-Policy Evaluationcited by 43
  2. 2021Local policy search with Bayesian optimizationcited by 38
  3. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 26
  4. 2024Learning Useful Representations of Recurrent Neural Network Weight Matricescited by 15
  5. 2022ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representationcited by 14
  6. 2022General Policy Evaluation and Improvement by Learning to Identify Few But Crucial Statescited by 10
  7. 2022Exploring through Random Curiosity with General Value Functionscited by 10
  8. 2022Goal-Conditioned Generators of Deep Policiescited by 9
  9. 2023Learning to Identify Critical States for Reinforcement Learning from Videoscited by 9
  10. 2022PAnDR: Fast Adaptation to New Environments from Offline Experiences via Decoupling Policy and Environment Representationscited by 6
  11. 2021Training and Generating Neural Networks in Compressed Weight Spacecited by 4
  12. 2022Learning One Abstract Bit at a Time Through Self-Invented Experiments Encoded as Neural Networkscited by 4
12 more
  1. 2025From Parameters to Behaviors: Unsupervised Compression of the Policy Spacecited by 4
  2. 2022Towards A Unified Policy Abstraction Theory and Representation Learning Approach in Markov Decision Processescited by 3
  3. 2024Compatible Gradient Approximations for Actor-Critic Algorithmscited by 2
  4. 2025Learning to Explore in Diverse Reward Settings via Temporal-Difference-Error Maximizationcited by 2
  5. 2026Shape Your Body: Value Gradients for Multi-Embodiment Robot Designcited by 2
  6. 2021A nearly Blackwell-optimal policy gradient methodcited by 1
  7. 2022Reinforcement Teachingcited by 1
  8. 2025Upside Down Reinforcement Learning with Policy Generatorscited by 1
  9. 2025Massively Scaling Explicit Policy-conditioned Value Functionscited by 1
  10. 2025K-Level Policy Gradients for Multi-Agent Reinforcement Learningcited by 1
  11. 2024Deterministic Exploration via Stationary Bellman Error Maximizationcited by 0
  12. 2025Functional Critics Are Essential for Actor-Critic: From Off-Policy Stability to Efficient Explorationcited by 0