Sample Complexity Bounds for Linear Constrained MDPs with a Generative Model

We consider infinite-horizon γ-discounted (linear) constrained Markov decision processes (CMDPs) where the objective is to find a policy that maximizes the expected cumulative reward subject to expected cumulative constraints. Given access to a generative model, we propose to solve CMDPs with a primal-dual framework that can leverage any black-box unconstrained MDP solver. For linear CMDPs with feature dimension d, we instantiate the framework by using mirror descent value iteration (textttMDVI)~citepkitamura2023regularization an example MDP solver. We provide sample complexity bounds for the resulting CMDP algorithm in two cases: (i) relaxed feasibility, where small constraint violations are allowed, and (ii) strict feasibility, where the output policy is required to exactly satisfy the constraint. For (i), we prove that the algorithm can return an ε-optimal policy with high probability by using tildeOleft(fracd2(1-γ)4ε2right) samples. For (ii), we show that the algorithm requires tildeOleft(fracd2(1-γ)6ε2zeta2right) samples, where zeta is the problem-dependent Slater constant that characterizes the size of the feasible region. Furthermore, we prove a lower-bound of Omegaleft(fracd2(1-γ)5ε2zeta2right) for the strict feasibility setting. We note that our upper bounds under both settings exhibit a near-optimal dependence on d, ε, and zeta. Finally, we instantiate our framework for tabular CMDPs and show that it can be used to recover near-optimal sample complexities in this setting.