mlsanityData to 7 Oct 2026

Behavior Prior Representation learning for Offline Reinforcement Learning

arXiv 2211.00863 ↗First posted 2 Nov 2022cs.LG cs.AIcited by 12

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) struggles in environments with rich and noisy inputs, where the agent only has access to a fixed dataset without environment interactions. Past works have proposed common workarounds based on the pre-training of state representations, followed by policy training. In this work, we introduce a simple, yet effective approach for learning state representations. Our method, Behavior Prior Representation (BPR), learns state representations with an easy-to-integrate objective based on behavior cloning of the dataset: we first learn a state representation by mimicking actions from the dataset, and then train a policy on top of the fixed representation, using any off-the-shelf Offline RL algorithm. Theoretically, we prove that BPR carries out performance guarantees when integrated into algorithms that have either policy improvement guarantees (conservative algorithms) or produce lower bounds of the policy values (pessimistic algorithms). Empirically, we show that BPR combined with existing state-of-the-art Offline RL algorithms leads to significant improvements across several offline control benchmarks. The code is available at urlhttps://github.com/bit1029public/offlinebpr.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 61 of its 84 references that are in the map. First posted on arXiv 2 Nov 2022.

Builds on

61 of 84 references.

Closest ancestors · 18

Same direction, newest first

  1. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  2. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization9 Dec
  3. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation21 Nov
  4. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  5. 2021BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning2 Oct
  6. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL16 Jun
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2021Representation Matters: Offline Pretraining for Sequential Decision Making11 Feb
  12. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
6 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020MOPO: Model-based Offline Policy Optimization27 May
  3. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  6. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 43

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  3. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  7. 2016End to End Learning for Self-Driving Carscited by 1,000
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  10. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  11. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  12. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
31 more
  1. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  2. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
  3. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  4. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  5. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  6. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  7. 2021Reinforcement Learning with Prototypical Representationscited by 179
  8. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  9. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  10. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  11. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  12. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  13. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  14. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  15. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  16. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  17. 2020Deep Reinforcement and InfoMax Learningcited by 79
  18. 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72
  19. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
  20. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  21. 2020Provable Representation Learning for Imitation Learning via Bi-level Optimizationcited by 41
  22. 2022Provable Benefits of Representational Transfer in Reinforcement Learningcited by 38
  23. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
  24. 2022On the Generalization of Representations in Reinforcement Learningcited by 28
  25. 2022Investigating the Properties of Neural Network Representations in Reinforcement Learningcited by 24
  26. 2019Safe Policy Improvement with Soft Baseline Bootstrappingcited by 22
  27. 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPscited by 22
  28. 2019Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environmentscited by 17
  29. 2022An Empirical Investigation of Representation Learning for Imitationcited by 15
  30. 2021SimSR: Simple Distance-based State Representation for Deep Reinforcement Learningcited by 14
  31. 2020Learning State Abstractions for Transfer in Continuous Controlcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 3; 2026: 3.

12 papers.

  1. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 30
  2. 2024Learning Latent Dynamic Robust Representations for World Modelscited by 20
  3. 2025Language Models Can Learn from Verbal Feedback Without Scalar Rewardscited by 17
  4. 2023Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learningcited by 10
  5. 2023Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learningcited by 7
  6. 2024Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learningcited by 4
  7. 2025Video-Enhanced Offline Reinforcement Learning: A Model-Based Approachcited by 1
  8. 2025SEG-Parking: Towards Safe, Efficient, and Generalizable Autonomous Parking via End-to-End Offline Reinforcement Learningcited by 1
  9. 2024Stable Offline Value Function Learning with Bisimulation-based Representationscited by 0
  10. 2026Expert Behavior Prior Reinforcement Learningcited by 0
  11. 2026Learning Reliable Parking Policies via Offline Reinforcement Learning with Quantized Action Representationscited by 0
  12. 2026Improving Online Reinforcement Learning via Bidirectional Behavior Prior Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Behavior Preference Regression for Offline Reinforcement Learning
  2. 2023Behavior Proximal Policy Optimization
  3. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
  4. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
  5. 2021Offline Reinforcement Learning with Soft Behavior Regularization
  6. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL
  7. 2024Efficient Offline Reinforcement Learning: First Imitate, then Improve
  8. 2024Offline Multi-task Transfer RL with Representational Penalization