mlsanityData to 14 Oct 2026

A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learning

arXiv 2306.07818 ↗First posted 13 Jun 2023cs.LG stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

Offline constrained reinforcement learning (RL) aims to learn a policy that maximizes the expected cumulative reward subject to constraints on expected cumulative cost using an existing dataset. In this paper, we propose Primal-Dual-Critic Algorithm (PDCA), a novel algorithm for offline constrained RL with general function approximation. PDCA runs a primal-dual algorithm on the Lagrangian function estimated by critics. The primal player employs a no-regret policy optimization oracle to maximize the Lagrangian estimate and the dual player acts greedily to minimize the Lagrangian estimate. We show that PDCA can successfully find a near saddle point of the Lagrangian, which is nearly optimal for the constrained RL problem. Unlike previous work that requires concentrability and a strong Bellman completeness assumption, PDCA only requires concentrability and realizability assumptions for sample-efficient learning.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 32 of its 60 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

32 of 60 references.

References · 32

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2021Convex optimizationcited by 2,515
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  6. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
  7. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  8. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  9. 2019Batch Policy Learning under Constraintscited by 244
  10. 2020Exploration-Exploitation in Constrained MDPscited by 149
  11. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  12. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
20 more
  1. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  2. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  3. 2020Batch Value-function Approximation with Only Realizabilitycited by 104
  4. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  5. 2021Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPscited by 80
  6. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 77
  7. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 71
  8. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  9. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 69
  10. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 69
  11. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  12. 2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 50
  13. 2021A Primal-Dual Approach to Constrained Markov Decision Processescited by 42
  14. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
  15. 2022Provably Efficient Model-Free Constrained RL with Linear Function Approximationcited by 31
  16. 2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
  17. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraintscited by 15
  18. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learningcited by 13
  19. 2022Bellman Residual Orthogonalization for Offline Reinforcement Learningcited by 10
  20. 2022A Near-Optimal Primal-Dual Method for Off-Policy Learning in CMDPcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 2.

9 papers.

  1. 2024Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learningcited by 10
  2. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  3. 2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learningcited by 4
  4. 2025Offline Reinforcement Learning via Inverse Optimizationcited by 3
  5. 2025Constrained Sliced Wasserstein Embeddingcited by 2
  6. 2026Offline Constrained RLHF with Multiple Preference Oraclescited by 1
  7. 2025GNN-DT: Graph Neural Network Enhanced Decision Transformer for Efficient Optimization in Dynamic Environmentscited by 0
  8. 2025Temporally smoothed incremental model-based heuristic dynamic programming for command-filtered cascaded online learning flight controlcited by 0
  9. 2026Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deploymentcited by 0