Towards Optimal Offline Reinforcement Learning
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
We study offline reinforcement learning problems with a long-run average reward objective. The state-action pairs generated by any fixed behavioral policy thus follow a Markov chain, and the em empirical state-action-next-state distribution satisfies a large deviations principle. We use the rate function of this large deviations principle to construct an uncertainty set for the unknown em true state-action-next-state distribution. We also construct a distribution shift transformation that maps any distribution in this uncertainty set to a state-action-next-state distribution of the Markov chain generated by a fixed evaluation policy, which may differ from the unknown behavioral policy. We prove that the worst-case average reward of the evaluation policy with respect to all distributions in the shifted uncertainty set provides, in a rigorous statistical sense, the least conservative estimator for the average reward under the unknown true distribution. This guarantee is available even if one has only access to one single trajectory of serially correlated state-action pairs. The emerging robust optimization problem can be viewed as a robust Markov decision process with a non-rectangular uncertainty set. We adapt an efficient policy gradient algorithm to solve this problem. Numerical experiments show that our methods compare favorably against state-of-the-art methods.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 64 references that are in the map. First posted on arXiv 15 Mar 2025.
Builds on
39 of 64 references.
Closest ancestors · 11
Same direction, newest first
- 2023Adversarial Model for Offline Reinforcement Learning21 Feb
- 2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage5 Feb
- 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation14 Sep
- 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
- 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning11 Apr
- 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
- 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism17 Oct
- 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
- 2021Continuous Doubly Constrained Batch Reinforcement Learning18 Feb
- 2021Fast Rates for the Regret of Offline Reinforcement Learning31 Jan
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
Other references · 28
- 2019Solving Rubik's Cube with a Robot Handcited by 734
- 2018Learning Dexterous In-Hand Manipulationcited by 687
- 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
- 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
- 2019Batch Policy Learning under Constraintscited by 244
- 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
- 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 183
- 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 117
- 2022Robust Reinforcement Learning using Offline Datacited by 96
- 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
- 2013Projection onto the probability simplex: An efficient algorithm with a simple proof, and an applicationcited by 67
- 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 42
16 more
- 2017From Data to Decisions: Distributionally Robust Optimization is Optimalcited by 38
- 2022Doubly Robust Distributionally Robust Off-Policy Evaluation and Learningcited by 34
- 2022Stochastic first-order methods for average-reward Markov decision processescited by 32
- 2018Deep Reinforcement Learning based Recommendation with Explicit User-Item Interactions Modelingcited by 31
- 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 31
- 2020Distributionally Robust Batch Contextual Banditscited by 20
- 2023Distributionally Robust Model-based Reinforcement Learning with Large State Spacescited by 17
- 2023Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Setscited by 16
- 2023Off-Policy Average Reward Actor-Critic with Deterministic Policy Searchcited by 10
- 2020Reliable Off-policy Evaluation for Reinforcement Learningcited by 9
- 2020A Pareto Dominance Principle for Data-Driven Optimizationcited by 7
- 2021Learning and Decision-Making with Data: Optimal Formulations and Phase Transitionscited by 7
- 2021Robust Generalization despite Distribution Shift via Minimum Discriminating Informationcited by 6
- 2021Distributionally Robust Optimization with Markovian Datacited by 4
- 2021Fast Projection Onto Convex Smooth Constraintscited by 3
- 2023Optimal Confidence Intervals via Moderate Deviations Theorycited by 1
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Robust Offline Reinforcement Learning for Non-Markovian Decision Processes
- 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approach
- 2023Selective Uncertainty Propagation in Offline RL
- 2024Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning
- 2024Upper and Lower Bounds for Distributionally Robust Off-Dynamics Reinforcement Learning
- 2025Sample Complexity of Distributionally Robust Average-Reward Reinforcement Learning
- 2025Optimal Single-Policy Sample Complexity and Transient Coverage for Average-Reward Offline RL
- 2024Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes