mlsanityData to 14 Oct 2026

Evaluation-Aware Reinforcement Learning

arXiv 2509.19464 ↗First posted 23 Sep 2025cs.AI cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Policy evaluation is a core component of many reinforcement learning (RL) algorithms and a critical tool for ensuring safe deployment of RL policies. However, existing policy evaluation methods often suffer from high variance or bias. To address these issues, we introduce Evaluation-Aware Reinforcement Learning (EvA-RL), a general policy learning framework that considers evaluation accuracy at train-time, as opposed to standard post-hoc policy evaluation methods. Specifically, EvA-RL directly optimizes policies for efficient and accurate evaluation, in addition to being performant. We provide an instantiation of EvA-RL and demonstrate through a combination of theoretical analysis and empirical results that EvA-RL effectively trades off between evaluation accuracy and expected return. Finally, we show that the evaluation-aware policy and the evaluation mechanism itself can be co-learned to mitigate this tradeoff, providing the evaluation benefits without significantly sacrificing policy performance. This work opens a new line of research that elevates reliable evaluation to a first-class principle in reinforcement learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 18 of its 32 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

18 of 32 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  5. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  6. 2018Learning Dexterous In-Hand Manipulationcited by 687
  7. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  8. 2019Challenges of Real-World Reinforcement Learningcited by 291
  9. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  10. 2019Batch Policy Learning under Constraintscited by 244
  11. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  12. 2022Discovered Policy Optimisationcited by 104
6 more
  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  2. 2020Chip Placement with Deep Reinforcement Learningcited by 83
  3. 2019The Value Function Polytope in Reinforcement Learningcited by 29
  4. 2017Data-Efficient Policy Evaluation Through Behavior Policy Searchcited by 26
  5. 2020Deep Reinforcement Learning for Closed-Loop Blood Glucose Controlcited by 26
  6. 2023Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluationcited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Multi-Policy Evaluation for Reinforcement Learning
  2. 2024Doubly Optimal Policy Evaluation for Reinforcement Learning
  3. 2026DataFlex-RL: An Evaluation Platform for RLVR Data Policies
  4. 2025Balancing Interpretability and Performance in Reinforcement Learning: An Adaptive Spectral Based Linear Approach
  5. 2026SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Data
  6. 2024Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning
  7. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review
  8. 2026Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluation