mlsanityData to 7 Oct 2026

False Correlation Reduction for Offline Reinforcement Learning

arXiv 2110.12468 ↗First posted 24 Oct 2021cs.LG cs.AIcited by 9

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) harnesses the power of massive datasets for resolving sequential decision problems. Most existing papers only discuss defending against out-of-distribution (OOD) actions while we investigate a broader issue, the false correlations between epistemic uncertainty and decision-making, an essential factor that causes suboptimality. In this paper, we propose falSe COrrelation REduction (SCORE) for offline RL, a practically effective and theoretically provable algorithm. We empirically show that SCORE achieves the SoTA performance with 3.1x acceleration on various tasks in a standard benchmark (D4RL). The proposed algorithm introduces an annealing behavior cloning regularizer to help produce a high-quality estimation of uncertainty which is critical for eliminating false correlations from suboptimality. Theoretically, we justify the rationality of the proposed method and prove its convergence to the optimal policy with a sublinear rate under mild assumptions.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 49 references that are in the map. First posted on arXiv 24 Oct 2021.

Builds on

37 of 49 references.

Closest ancestors · 16

Same direction, newest first

  1. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  2. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  5. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  7. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  8. 2020Batch Value-function Approximation with Only Realizability11 Aug
  9. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  10. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  11. 2020MOPO: Model-based Offline Policy Optimization27 May
  12. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
4 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  3. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  4. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 21

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  7. 2016Deep Exploration via Bootstrapped DQNcited by 518
  8. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  9. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  10. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  11. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  12. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
9 more
  1. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  2. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  3. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  4. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  5. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  6. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  7. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  8. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  9. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 4; 2025: 1; 2026: 1.

9 papers.

  1. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  2. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  3. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
  4. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  5. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  6. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 7
  7. 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentationcited by 2
  8. 2024Cooperative Sentiment Agents for Multimodal Sentiment Analysiscited by 1
  9. 2026Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  2. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction
  3. 2024CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learning
  4. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
  5. 2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning
  6. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding
  7. 2023Selective Uncertainty Propagation in Offline RL
  8. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes