Data to 7 Oct 2026

Conservative Offline Distributional Reinforcement Learning

arXiv 2107.06106 ↗First posted 12 Jul 2021cs.LGcited by 78

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Many reinforcement learning (RL) problems in practice are offline, learning purely from observational data. A key challenge is how to ensure the learned policy is safe, which requires quantifying the risk associated with different actions. In the online setting, distributional RL algorithms do so by learning the distribution over returns (i.e., cumulative rewards) instead of the expected return; beyond quantifying risk, they have also been shown to learn better representations for planning. We propose Conservative Offline Distributional Actor Critic (CODAC), an offline RL algorithm suitable for both risk-neutral and risk-averse domains. CODAC adapts distributional RL to the offline setting by penalizing the predicted quantiles of the return for out-of-distribution actions. We prove that CODAC learns a conservative return distribution -- in particular, for finite MDPs, CODAC converges to an uniform lower bound on the quantiles of the return distribution; our proof relies on a novel analysis of the distributional Bellman operator. In our experiments, on two challenging robot navigation tasks, CODAC successfully learns risk-averse policies using offline data collected purely from risk-neutral agents. Furthermore, CODAC is state-of-the-art on the D4RL MuJoCo benchmark in terms of both expected and risk-sensitive performance.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 52 references that are in the map. First posted on arXiv 12 Jul 2021.

Builds on

30 of 52 references.

Closest ancestors · 7

Same direction, newest first

2021Risk-Averse Offline Reinforcement Learning10 Feb
2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
2020MOPO: Model-based Offline Policy Optimization27 May
2020MOReL : Model-Based Offline Reinforcement Learning12 May
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 23

2016Layer Normalizationcited by 5,579
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2015Continuous control with deep reinforcement learningcited by 3,279
2013Playing Atari with Deep Reinforcement Learningcited by 3,211
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
2017A Distributional Perspective on Reinforcement Learningcited by 743
2017Distributional Reinforcement Learning with Quantile Regressioncited by 464

11 more

2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
2018Distributed Distributional Deterministic Policy Gradientscited by 195
2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 87
2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
2019Worst Cases Policy Gradientscited by 52
2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 17; 2023: 20; 2024: 14; 2025: 11; 2026: 13.

78 papers.

Sort:

By area:

2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 30
2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
2022Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 26
2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learningcited by 23
2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
2022Addressing Optimism Bias in Sequence Modeling for Reinforcement Learningcited by 22

38 more

2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 21
2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 19
2022Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Boundscited by 16
2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 15
2021Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learningcited by 13
2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learningcited by 13
2022Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discoverycited by 12
2025Value Flowscited by 12
2024Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learningcited by 10
2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 9
2022Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probabilitycited by 9
2022Distributional Reinforcement Learning with Regularized Wasserstein Losscited by 8
2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
2024The Central Role of the Loss Function in Reinforcement Learningcited by 8
2023Risk-Aware Reinforcement Learning through Optimal Transport Theorycited by 7
2023Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learningcited by 6
2023Decoupled Prioritized Resampling for Offline RLcited by 6
2025Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimationcited by 6
2023Constrained Reinforcement Learning using Distributional Representation for Trustworthy Quadrotor UAV Tracking Controlcited by 5
2023Robotic Manipulation Datasets for Offline Compositional Reinforcement Learningcited by 5
2023Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theorycited by 5
2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learningcited by 5
2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learningcited by 5
2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 5
2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RLcited by 5
2021Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learningcited by 4
2022How Does Return Distribution in Distributional Reinforcement Learning Help Optimization?cited by 4
2023Robustified Learning for Online Optimization with Memory Costscited by 4
2025Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Datasetcited by 4
2023OER: Offline Experience Replay for Continual Offline Reinforcement Learningcited by 3
2023Robust Learning for Smoothed Online Convex Optimization with Feedback Delaycited by 3
2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learningcited by 3
2022Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxationcited by 2
2023Is Risk-Sensitive Reinforcement Learning Properly Resolved?cited by 2
2024Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learningcited by 2

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way