mlsanityData to 7 Oct 2026

Instabilities of Offline RL with Pre-Trained Neural Representation

arXiv 2103.04947 ↗First posted 8 Mar 2021cs.LG cs.AI math.OCcited by 38

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In offline reinforcement learning (RL), we seek to utilize offline data to evaluate (or learn) policies in scenarios where the data are collected from a distribution that substantially differs from that of the target policy to be evaluated. Recent theoretical advances have shown that such sample-efficient offline RL is indeed possible provided certain strong representational conditions hold, else there are lower bounds exhibiting exponential error amplification (in the problem horizon) unless the data collection distribution has only a mild distribution shift relative to the target policy. This work studies these issues from an empirical perspective to gauge how stable offline RL methods are. In particular, our methodology explores these ideas when using features from pre-trained neural networks, in the hope that these representations are powerful enough to permit sample efficient offline RL. Through extensive experiments on a range of tasks, we see that substantial error amplification does occur even when using such pre-trained representations (trained on the same task itself); we find offline RL is stable only under extremely mild distribution shift. The implications of these results, both from a theoretical and an empirical perspective, are that successful offline RL (where we seek to go beyond the low distribution shift regime) requires substantially stronger conditions beyond those which suffice for successful supervised learning.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 47 of its 79 references that are in the map. First posted on arXiv 8 Mar 2021.

Builds on

47 of 79 references.

Closest ancestors · 12

Same direction, newest first

  1. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  2. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  3. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  4. 2020Batch Value-function Approximation with Only Realizability11 Aug
  5. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020MOPO: Model-based Offline Policy Optimization27 May
  8. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  12. 2019An Optimistic Perspective on Offline Reinforcement Learning10 Jul

Other references · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2018Deep contextualized word representationscited by 2,285
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2014How transferable are features in deep neural networks?cited by 1,216
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  9. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  10. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  11. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  12. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
23 more
  1. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  2. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  3. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  4. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  5. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  6. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  7. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  8. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  9. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  10. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  11. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  12. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  13. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  14. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  15. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  16. 2017Towards Generalization and Simplicity in Continuous Controlcited by 65
  17. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  18. 2020Semantic Visual Navigation by Watching YouTube Videoscited by 60
  19. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  20. 2019Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Samplingcited by 36
  21. 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
  22. 2020A Variant of the Wang-Foster-Kakade Lower Bound for the Discounted Settingcited by 24
  23. 2020Minimax Value Interval for Off-Policy Evaluation and Policy Optimizationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 6; 2023: 12; 2024: 7; 2025: 2; 2026: 2.

38 papers.

Sort:

By area:

  1. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  2. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  3. 2021Offline RL Without Off-Policy Evaluationcited by 150
  4. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  5. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  6. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  7. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 69
  8. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  9. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  10. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  11. 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
  12. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
26 more
  1. 2024Hybrid Inverse Reinforcement Learningcited by 31
  2. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  3. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functionscited by 17
  4. 2023Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Banditscited by 14
  5. 2023Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregressioncited by 14
  6. 2023Why Target Networks Stabilise Temporal Difference Methodscited by 13
  7. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
  8. 2022A Complete Characterization of Linear Estimators for Offline Policy Evaluationcited by 7
  9. 2022Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstractioncited by 7
  10. 2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
  11. 2023Model-based Offline Reinforcement Learning with Local Misspecificationcited by 5
  12. 2023Bi-Level Offline Policy Optimization with Limited Explorationcited by 5
  13. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5
  14. 2023STEEL: Singularity-aware Reinforcement Learningcited by 3
  15. 2023State-Action Similarity-Based Representations for Off-Policy Evaluationcited by 3
  16. 2024Off-policy Evaluation with Deeply-abstracted Statescited by 2
  17. 2025Improving the Data-efficiency of Reinforcement Learning by Warm-starting with LLMcited by 2
  18. 2021The Impact of Data Distribution on Q-learning with Function Approximationcited by 1
  19. 2022Optimal Estimation of Off-Policy Policy Gradient via Double Fitted Iterationcited by 1
  20. 2023Offline Reinforcement Learning with Additional Covering Distributionscited by 1
  21. 2023Stackelberg Batch Policy Learningcited by 1
  22. 2024Enhancing Quantum Memory Lifetime with Measurement-Free Local Error Correction and Reinforcement Learningcited by 1
  23. 2026A Unifying View of Coverage in Linear Off-Policy Evaluationcited by 1
  24. 2024Misspecified Q-Learning with Sparse Linear Function Approximation: Tight Bounds on Approximation Errorcited by 0
  25. 2024Stable Offline Value Function Learning with Bisimulation-based Representationscited by 0
  26. 2026Continuous-time reinforcement learning: ellipticity enables model-free value function approximationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  2. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  3. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage
  4. 2021A Minimalist Approach to Offline Reinforcement Learning
  5. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  6. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  7. 2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints
  8. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model