# A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs

arXiv: https://arxiv.org/abs/2402.04493 · HTML: https://mlsanity.com/paper/2402.04493/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 7 Feb 2024.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Built on by (most cited first, 5 of 5)

- [Offline RL via Feature-Occupancy Gradient Ascent](https://mlsanity.com/paper/2405.13755/) (2024)
- [Offline Constrained Reinforcement Learning under Partial Data Coverage](https://mlsanity.com/paper/2505.17506/) (2025)
- [Consensus Based Stochastic Control](https://mlsanity.com/paper/2501.17801/) (2025)
- [A Two-Timescale Primal-Dual Framework for Reinforcement Learning via Online Dual Variable Guidance](https://mlsanity.com/paper/2505.04494/) (2025)
- [LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy](https://mlsanity.com/paper/2602.17312/) (2026)

## Abstract (arXiv)

We study offline reinforcement learning (RL) with linear MDPs under the infinite-horizon discounted setting which aims to learn a policy that maximizes the expected discounted cumulative reward using a pre-collected dataset. Existing algorithms for this setting either require a uniform data coverage assumptions or are computationally inefficient for finding an ε-optimal policy with O(ε-2) sample complexity. In this paper, we propose a primal dual algorithm for offline RL with linear MDPs in the infinite-horizon discounted setting. Our algorithm is the first computationally efficient algorithm in this setting that achieves sample complexity of O(ε-2) with partial data coverage assumption. Our work is an improvement upon a recent work that requires O(ε-4) samples. Moreover, we extend our algorithm to work in the offline constrained RL setting that enforces constraints on additional reward signals.