mlsanityData to 14 Oct 2026

State-Action Similarity-Based Representations for Off-Policy Evaluation

arXiv 2310.18409 ↗First posted 27 Oct 2023cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning, off-policy evaluation (OPE) is the problem of estimating the expected return of an evaluation policy given a fixed dataset that was collected by running one or more different policies. One of the more empirically successful algorithms for OPE has been the fitted q-evaluation (FQE) algorithm that uses temporal difference updates to learn an action-value function, which is then used to estimate the expected return of the evaluation policy. Typically, the original fixed dataset is fed directly into FQE to learn the action-value function of the evaluation policy. Instead, in this paper, we seek to enhance the data-efficiency of FQE by first transforming the fixed dataset using a learned encoder, and then feeding the transformed dataset into FQE. To learn such an encoder, we introduce an OPE-tailored state-action behavioral similarity metric, and use this metric and the fixed dataset to learn an encoder that models this metric. Theoretically, we show that this metric allows us to bound the error in the resulting OPE estimate. Empirically, we show that other state-action similarity metrics lead to representations that cannot represent the action-value function of the evaluation policy, and that our state-action representation method boosts the data-efficiency of FQE and lowers OPE error relative to other OPE-based representation learning methods on challenging OPE tasks. We also empirically show that the learned representations significantly mitigate divergence of FQE under varying distribution shifts. Our code is available here: https://github.com/Badger-RL/ROPE.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 27 Oct 2023.

Builds on

28 of 47 references.

References · 28

  1. 2016Layer Normalizationcited by 5,669
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  7. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  8. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  9. 2019Batch Policy Learning under Constraintscited by 244
  10. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  11. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
  12. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
16 more
  1. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  2. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  3. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  4. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  5. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  6. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  7. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 91
  8. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  9. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  10. 2021Towards Robust Bisimulation Metric Learningcited by 49
  11. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  12. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  13. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 33
  14. 2021Offline Reinforcement Learning with Pseudometric Learningcited by 28
  15. 2022Learning Bellman Complete Representations for Offline Policy Evaluationcited by 17
  16. 2022Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstractioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 0; 2026: 1.

4 papers.

  1. 2024Off-policy Evaluation with Deeply-abstracted Statescited by 4
  2. 2024Stable Offline Value Function Learning with Bisimulation-based Representationscited by 0
  3. 2024Conceptual Belief-Informed Reinforcement Learningcited by 0
  4. 2026What a Data Release Must Preserve for Off-Policy Evaluationcited by 0