# Policy Evaluation Networks

arXiv: https://arxiv.org/abs/2002.11833 · HTML: https://mlsanity.com/paper/2002.11833/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2020.

## Built on by (most cited first, 28 of 28)

- [Permutation Equivariant Neural Functionals](https://mlsanity.com/paper/2302.14040/) (2023)
- [A Survey of Temporal Credit Assignment in Deep Reinforcement Learning](https://mlsanity.com/paper/2312.01072/) (2023)
- [Universal Off-Policy Evaluation](https://mlsanity.com/paper/2104.12820/) (2021)
- [Active Offline Policy Selection](https://mlsanity.com/paper/2106.10251/) (2021)
- [Hypernetworks for Zero-shot Transfer in Reinforcement Learning](https://mlsanity.com/paper/2211.15457/) (2022)
- [Monomial Matrix Group Equivariant Neural Functional Networks](https://mlsanity.com/paper/2409.11697/) (2024)
- [Learning Useful Representations of Recurrent Neural Network Weight Matrices](https://mlsanity.com/paper/2403.11998/) (2024)
- [General Policy Evaluation and Improvement by Learning to Identify Few But Crucial States](https://mlsanity.com/paper/2207.01566/) (2022)
- [Exploring through Random Curiosity with General Value Functions](https://mlsanity.com/paper/2211.10282/) (2022)
- [Value function estimation using conditional diffusion models for control](https://mlsanity.com/paper/2306.07290/) (2023)
- [Goal-Conditioned Generators of Deep Policies](https://mlsanity.com/paper/2207.01570/) (2022)
- [Learning to Identify Critical States for Reinforcement Learning from Videos](https://mlsanity.com/paper/2308.07795/) (2023)
- [First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization](https://mlsanity.com/paper/2205.12381/) (2022)
- [The Geometry of Robust Value Functions](https://mlsanity.com/paper/2201.12929/) (2022)
- [PAnDR: Fast Adaptation to New Environments from Offline Experiences via Decoupling Policy and Environment Representations](https://mlsanity.com/paper/2204.02877/) (2022)
- [Meta-Value Learning: a General Framework for Learning with Learning Awareness](https://mlsanity.com/paper/2307.08863/) (2023)
- [Training and Generating Neural Networks in Compressed Weight Space](https://mlsanity.com/paper/2112.15545/) (2021)
- [Learning One Abstract Bit at a Time Through Self-Invented Experiments Encoded as Neural Networks](https://mlsanity.com/paper/2212.14374/) (2022)
- [Towards A Unified Policy Abstraction Theory and Representation Learning Approach in Markov Decision Processes](https://mlsanity.com/paper/2209.07696/) (2022)
- [Shape Your Body: Value Gradients for Multi-Embodiment Robot Design](https://mlsanity.com/paper/2606.00702/) (2026)
- [A nearly Blackwell-optimal policy gradient method](https://mlsanity.com/paper/2105.13609/) (2021)
- [Reinforcement Teaching](https://mlsanity.com/paper/2204.11897/) (2022)
- [Upside Down Reinforcement Learning with Policy Generators](https://mlsanity.com/paper/2501.16288/) (2025)
- [Massively Scaling Explicit Policy-conditioned Value Functions](https://mlsanity.com/paper/2502.11949/) (2025)
- [Learning Policy Representations for Steerable Behavior Synthesis](https://mlsanity.com/paper/2601.22350/) (2026)
- [Quasi-Equivariant Metanetworks](https://mlsanity.com/paper/2604.23720/) (2026)
- [Deterministic Exploration via Stationary Bellman Error Maximization](https://mlsanity.com/paper/2410.23840/) (2024)
- [Functional Critics Are Essential for Actor-Critic: From Off-Policy Stability to Efficient Exploration](https://mlsanity.com/paper/2509.22964/) (2025)

## Abstract (arXiv)

Many reinforcement learning algorithms use value functions to guide the search for better policies. These methods estimate the value of a single policy while generalizing across many states. The core idea of this paper is to flip this convention and estimate the value of many policies, for a single set of states. This approach opens up the possibility of performing direct gradient ascent in policy space without seeing any new data. The main challenge for this approach is finding a way to represent complex policies that facilitates learning and generalization. To address this problem, we introduce a scalable, differentiable fingerprinting mechanism that retains essential policy information in a concise embedding. Our empirical results demonstrate that combining these three elements (learned Policy Evaluation Network, policy fingerprints, gradient ascent) can produce policies that outperform those that generated the training data, in zero-shot manner.