A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
We study offline reinforcement learning (RL) with linear MDPs under the infinite-horizon discounted setting which aims to learn a policy that maximizes the expected discounted cumulative reward using a pre-collected dataset. Existing algorithms for this setting either require a uniform data coverage assumptions or are computationally inefficient for finding an ε-optimal policy with O(ε-2) sample complexity. In this paper, we propose a primal dual algorithm for offline RL with linear MDPs in the infinite-horizon discounted setting. Our algorithm is the first computationally efficient algorithm in this setting that achieves sample complexity of O(ε-2) with partial data coverage assumption. Our work is an improvement upon a recent work that requires O(ε-4) samples. Moreover, we extend our algorithm to work in the offline constrained RL setting that enforces constraints on additional reward signals.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. First posted on arXiv 7 Feb 2024.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2024: 1; 2025: 2; 2026: 1.
4 papers.
- 2024Offline RL via Feature-Occupancy Gradient Ascentcited by 2
- 2025Offline Constrained Reinforcement Learning under Partial Data Coveragecited by 1
- 2025A Two-Timescale Primal-Dual Framework for Reinforcement Learning via Online Dual Variable Guidancecited by 0
- 2026LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchycited by 0
- 2026LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy19 Feb 2026
- 2025Offline Constrained Reinforcement Learning under Partial Data Coverage23 May 2025
- 2025A Two-Timescale Primal-Dual Framework for Reinforcement Learning via Online Dual Variable Guidance7 May 2025
- 2024Offline RL via Feature-Occupancy Gradient Ascent22 May 2024
Similar, not cited
Close in wording, no citation either way
- 2023Offline Primal-Dual Reinforcement Learning for Linear MDPs
- 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability
- 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints
- 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism
- 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction
- 2025Optimal Single-Policy Sample Complexity and Transient Coverage for Average-Reward Offline RL
- 2023Offline Reinforcement Learning with Additional Covering Distributions
- 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning