mlsanityData to 7 Oct 2026

Offline Reinforcement Learning with Pseudometric Learning

arXiv 2103.01948 ↗First posted 2 Mar 2021cs.LGcited by 28

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement Learning methods seek to learn a policy from logged transitions of an environment, without any interaction. In the presence of function approximation, and under the assumption of limited coverage of the state-action space of the environment, it is necessary to enforce the policy to visit state-action pairs close to the support of logged transitions. In this work, we propose an iterative procedure to learn a pseudometric (closely related to bisimulation metrics) from logged transitions, and use it to define this notion of closeness. We show its convergence and extend it to the function approximation setting. We then use this pseudometric to define a new lookup based bonus in an actor-critic algorithm: PLOFF. This bonus encourages the actor to stay close, in terms of the defined pseudometric, to the support of logged transitions. Finally, we evaluate the method on hand manipulation and locomotion tasks.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 77 references that are in the map. First posted on arXiv 2 Mar 2021.

Builds on

33 of 77 references.

Closest ancestors · 7

Same direction, newest first

  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020MOPO: Model-based Offline Policy Optimization27 May
  3. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  6. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 26

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016OpenAI Gymcited by 1,684
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2015Prioritized Experience Replaycited by 1,070
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  11. 2017Deep Reinforcement Learning that Matterscited by 727
  12. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
14 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  2. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  3. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  4. 2019Challenges of Real-World Reinforcement Learningcited by 282
  5. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  6. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  7. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  8. 2020Model-Based Offline Planningcited by 126
  9. 2020Primal Wasserstein Imitation Learningcited by 120
  10. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  11. 2020Plannable Approximations to MDP Homomorphisms: Equivariance under Actionscited by 61
  12. 2021Metrics and continuity in reinforcement learningcited by 26
  13. 2019Safe Policy Improvement with Soft Baseline Bootstrappingcited by 22
  14. 2019Safe Policy Improvement with an Estimated Baseline Policycited by 16

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 9; 2023: 6; 2024: 4; 2025: 3.

28 papers.

Sort:

By area:

  1. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  2. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  3. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  4. 2021Offline Reinforcement Learning as Anti-Explorationcited by 44
  5. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  6. 2021RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learningcited by 31
  7. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  8. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  9. 2022Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptationcited by 28
  10. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RLcited by 21
  11. 2023Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Drivingcited by 16
  12. 2022Model-based Trajectory Stitching for Improved Offline Reinforcement Learningcited by 15
16 more
  1. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
  2. 2022State Representation Learning for Goal-Conditioned Reinforcement Learningcited by 11
  3. 2021Modified DDPG car-following model with a real-world human driving experience with CARLA simulatorcited by 9
  4. 2022Model-based trajectory stitching for improved behavioural cloning and its applicationscited by 7
  5. 2024Solving Continual Offline Reinforcement Learning with Decision Transformercited by 6
  6. 2023OER: Offline Experience Replay for Continual Offline Reinforcement Learningcited by 3
  7. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularizationcited by 3
  8. 2023State-Action Similarity-Based Representations for Off-Policy Evaluationcited by 3
  9. 2025Wavelet Fourier Diffuser: Frequency-Aware Diffusion Model for Reinforcement Learningcited by 3
  10. 2023K-Nearest-Neighbor Resampling for Off-Policy Evaluation in Stochastic Controlcited by 2
  11. 2023Deep Generative Models for Decision-Making and Controlcited by 1
  12. 2024State-Constrained Offline Reinforcement Learningcited by 1
  13. 2024Strategically Conservative Q-Learningcited by 1
  14. 2024Conceptual Belief-Informed Reinforcement Learningcited by 0
  15. 2025Learning The Minimum Action Distancecited by 0
  16. 2025A Theoretical Analysis of State Similarity Between Markov Decision Processescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies
  2. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  3. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
  4. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Models
  5. 2024Offline Reinforcement Learning from Datasets with Structured Non-Stationarity
  6. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees
  7. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  8. 2026Multi-step Proximal Policy Improvement in Offline Reinforcement Learning