mlsanityData to 14 Oct 2026

Offline Constrained Reinforcement Learning under Partial Data Coverage

arXiv 2505.17506 ↗First posted 23 May 2025stat.ML cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We study offline constrained reinforcement learning with general function approximation in discounted constrained Markov decision processes. Prior methods either require full data coverage for evaluating intermediate policies, lack oracle efficiency, or requires the knowledge of data-generating distribution for policy extraction. We propose PDOCRL, an oracle-efficient primal-dual algorithm based on a decomposed linear-programming formulation that makes the policy an explicit optimization variable. This avoids policy extraction that requires the knowledge of data-generating distribution, and only uses standard policy-optimization, online linear-optimization, and linear-minimization oracles. We show that saddle-point formulations using general function approximation can have spurious saddle points even when an optimal solution is realizable, and identify a stronger realizability condition under which every restricted saddle point is optimal. Under this condition and partial coverage of an optimal policy, PDOCRL returns a near-optimal, near-feasible policy with a (widetildemathcal O(ε-2)) sample guarantee, without access to the data-generating distribution. Empirically, PDOCRL is competitive with strong baselines on standard offline constrained RL benchmarks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

32 of 44 references.

References · 32

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  3. 2019A Theory of Regularized Markov Decision Processescited by 286
  4. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  5. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  6. 2019Batch Policy Learning under Constraintscited by 244
  7. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  8. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  9. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  10. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
  11. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  12. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
20 more
  1. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 119
  2. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  3. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  4. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  5. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  6. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  7. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  8. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 59
  9. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 36
  10. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
  11. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  12. 2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
  13. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPscited by 17
  14. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learningcited by 13
  15. 2024Sample-Efficient Constrained Reinforcement Learning with General Parameterizationcited by 11
  16. 2022Efficient Global Planning in Large MDPs via Stochastic Primal-Dual Optimizationcited by 8
  17. 2022Policy-based Primal-Dual Methods for Concave CMDP with Variance Reductioncited by 5
  18. 2024A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPscited by 5
  19. 2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learningcited by 4
  20. 2024Offline RL via Feature-Occupancy Gradient Ascentcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Offline Constrained RLHF with Multiple Preference Oraclescited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints
  2. 2023Offline Reinforcement Learning with Additional Covering Distributions
  3. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
  4. 2022On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation
  5. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism
  6. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction
  7. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  8. 2025Online Optimization for Offline Safe Reinforcement Learning