Offline RL via Feature-Occupancy Gradient Ascent
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
We study offline Reinforcement Learning in large infinite-horizon discounted Markov Decision Processes (MDPs) when the reward and transition models are linearly realizable under a known feature map. Starting from the classic linear-program formulation of the optimal control problem in MDPs, we develop a new algorithm that performs a form of gradient ascent in the space of feature occupancies, defined as the expected feature vectors that can potentially be generated by executing policies in the environment. We show that the resulting simple algorithm satisfies strong computational and sample complexity guarantees, achieved under the least restrictive data coverage assumptions known in the literature. In particular, we show that the sample complexity of our method scales optimally with the desired accuracy level and depends on a weak notion of coverage that only requires the empirical feature covariance matrix to cover a single direction in the feature space (as opposed to covering a full subspace). Additionally, our method is easy to implement and requires no prior knowledge of the coverage ratio (or even an upper bound on it), which altogether make it the strongest known algorithm for this setting to date.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 16 of its 42 references that are in the map. First posted on arXiv 22 May 2024.
Builds on
16 of 42 references.
Closest ancestors · 9
Same direction, newest first
- 2024A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs7 Feb
- 2023Offline Primal-Dual Reinforcement Learning for Linear MDPs22 May
- 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian1 Nov
- 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning11 Apr
- 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
- 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
- 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
- 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
Other references · 7
- 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 180
- 2020A Unifying View of Optimism in Episodic Reinforcement Learningcited by 50
- 2020Logistic Q-Learningcited by 34
- 2023Unconstrained Online Learning with Unbounded Lossescited by 25
- 2024Dealing with unbounded gradients in stochastic saddle-point optimizationcited by 15
- 2022Efficient Global Planning in Large MDPs via Stochastic Primal-Dual Optimizationcited by 8
- 2022Confident Approximate Policy Iteration for Efficient Local Planning in qπ-realizable MDPscited by 8
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
- 2026Generative Modeling by Value-Driven Transport21 May 2026
- 2025Offline Constrained Reinforcement Learning under Partial Data Coverage23 May 2025
Similar, not cited
Close in wording, no citation either way
- 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints
- 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction
- 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism
- 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
- 2023Offline Reinforcement Learning with Additional Covering Distributions
- 2025Optimal Single-Policy Sample Complexity and Transient Coverage for Average-Reward Offline RL
- 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
- 2020MOPO: Model-based Offline Policy Optimization