mlsanityData to 7 Oct 2026

Survival Instinct in Offline Reinforcement Learning

arXiv 2306.03286 ↗First posted 5 Jun 2023cs.LG cs.AIcited by 20

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We present a novel observation about the behavior of offline reinforcement learning (RL) algorithms: on many benchmark datasets, offline RL can produce well-performing and safe policies even when trained with "wrong" reward labels, such as those that are zero everywhere or are negatives of the true rewards. This phenomenon cannot be easily explained by offline RL's return maximization objective. Moreover, it gives offline RL a degree of robustness that is uncharacteristic of its online RL counterparts, which are known to be sensitive to reward design. We demonstrate that this surprising robustness property is attributable to an interplay between the notion of pessimism in offline RL algorithms and certain implicit biases in common data collection practices. As we prove in this work, pessimism endows the agent with a "survival instinct", i.e., an incentive to stay within the data support in the long term, while the limited and biased data coverage further constrains the set of survival policies. Formally, given a reward class -- which may not even contain the true reward -- we identify conditions on the training data distribution that enable offline RL to learn a near-optimal and safe policy from any reward within the class. We argue that the survival instinct should be taken into account when interpreting results from existing offline RL benchmarks and when creating future ones. Our empirical and theoretical results suggest a new paradigm for RL, whereby an agent is nudged to learn a desirable behavior with imperfect reward but purposely biased data coverage.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 59 of its 72 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

59 of 72 references.

Closest ancestors · 28

Same direction, newest first

  1. 2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learning27 Feb
  2. 2023Adversarial Model for Offline Reinforcement Learning21 Feb
  3. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewards3 Feb
  4. 2022ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Data8 Nov
  5. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation14 Sep
  6. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
  7. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  8. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  9. 2022COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks16 Mar
  10. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  11. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  12. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learning3 Feb
16 more
  1. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning31 Jan
  2. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation21 Nov
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  5. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  6. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Corruption-Robust Offline Reinforcement Learning11 Jun
  9. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  13. 2020MOPO: Model-based Offline Policy Optimization27 May
  14. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  15. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  16. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 31

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  3. 2017Constrained Policy Optimizationcited by 928
  4. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  5. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  6. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  7. 2017Inverse Reward Designcited by 246
  8. 2019Batch Policy Learning under Constraintscited by 242
  9. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  10. 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 109
  11. 2017AI Safety Gridworldscited by 108
  12. 2022Robust Reinforcement Learning using Offline Datacited by 95
19 more
  1. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
  2. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  3. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  4. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 75
  5. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  6. 2021Off-Policy Imitation Learning from Observationscited by 70
  7. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  8. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  9. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  10. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 45
  11. 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
  12. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  13. 2018Policy Transfer with Strategy Optimizationcited by 41
  14. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  15. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
  16. 2022Byzantine-Robust Online and Offline Distributed Reinforcement Learningcited by 13
  17. 2023A Strong Baseline for Batch Imitation Learningcited by 5
  18. 2021Uniformly Conservative Exploration in Reinforcement Learningcited by 3
  19. 2023Provable Reset-free Reinforcement Learning by No-Regret Reductioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 10; 2025: 6; 2026: 2.

20 papers.

Sort:

By area:

  1. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  2. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
  3. 2023Unsupervised Behavior Extraction via Random Intent Priorscited by 13
  4. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  5. 2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptionscited by 8
  6. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
  7. 2024Offline Reinforcement Learning for Learning to Dispatch for Job Shop Schedulingcited by 6
  8. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
  9. 2024Universal Black-Box Reward Poisoning Attack against Offline Reinforcement Learningcited by 4
  10. 2024Adversarially Robust Decision Transformercited by 4
  11. 2024Hindsight Preference Learning for Offline Preference-based Reinforcement Learningcited by 3
  12. 2025CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queriescited by 3
8 more
  1. 2025Adversarial Policy Optimization for Offline Preference-based Reinforcement Learningcited by 2
  2. 2025ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learningcited by 2
  3. 2024SaVeR: Optimal Data Collection Strategy for Safe Policy Evaluation in Tabular MDPcited by 1
  4. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learningcited by 1
  5. 2025TROFI: Trajectory-Ranked Offline Inverse Reinforcement Learningcited by 0
  6. 2025Generalizing Beyond Suboptimality: Offline Reinforcement Learning Learns Effective Scheduling through Random Solutionscited by 0
  7. 2026OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Explorationcited by 0
  8. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  2. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning
  3. 2023Robust Offline Reinforcement learning with Heavy-Tailed Rewards
  4. 2025Semi-pessimistic Reinforcement Learning
  5. 2021Instabilities of Offline RL with Pre-Trained Neural Representation
  6. 2020Offline Reinforcement Learning Hands-On
  7. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
  8. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learning