# Near-Optimal Reinforcement Learning for Constrained Recurrence Objectives

arXiv: https://arxiv.org/abs/2511.19849 · HTML: https://mlsanity.com/paper/2511.19849/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 9 of its 20 references that are in the map.
- First posted on arXiv 25 Nov 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 9 of 9)

- [Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learning](https://mlsanity.com/paper/1909.07299/) (2019)
- [Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraints](https://mlsanity.com/paper/1404.7073/) (2014)
- [Near-Optimal Sample Complexity Bounds for Constrained MDPs](https://mlsanity.com/paper/2206.06270/) (2022)
- [Policy Optimization with Linear Temporal Logic Constraints](https://mlsanity.com/paper/2206.09546/) (2022)
- [Span-Based Optimal Sample Complexity for Weakly Communicating and General Average Reward MDPs](https://mlsanity.com/paper/2403.11477/) (2024)
- [Optimal Probabilistic Motion Planning with Potential Infeasible LTL Constraints](https://mlsanity.com/paper/2007.14325/) (2020)
- [A PAC Learning Algorithm for LTL and Omega-regular Objectives in MDPs](https://mlsanity.com/paper/2310.12248/) (2023)
- [LTL-Constrained Policy Optimization with Cycle Experience Replay](https://mlsanity.com/paper/2404.11578/) (2024)
- [Near-Optimal Sample Complexity Bounds for Constrained Average-Reward MDPs](https://mlsanity.com/paper/2509.16586/) (2025)

## Abstract (arXiv)

Recurrence objectives, where a target region must be visited infinitely often, are a fundamental class of specifications for Markov decision processes (MDPs) and form the core of ω-regular and linear temporal logic (LTL) objectives. We study constrained recurrence objectives, a natural extension of recurrence objectives with probabilistic constraints capable of modelling safety or fairness requirements. We first study the structure of optimal policies, showing that constrained recurrence requires different policy classes than those sufficient for other constrained MDP formalisms. In particular, we show that every feasible instance admits an optimal mixture of two stochastic stationary policies, as well as an optimal mixture of two deterministic stationary policies over a one-bit augmented MDP. We then study the generative-model reinforcement learning setting and propose an algorithm that first identifies the maximal end-component decomposition of the MDP, then reduces constrained recurrence to a constrained average reward problem for a collapsed MDP. Moreover, we establish a tildeO(1/p+B/varepsilon2) sample complexity guarantee per state-action pair, where p bounds certain non-zero transition probabilities and B bounds transient time. Finally, we prove a nearly matching lower bound, showing that 1/p dependence, unlike the average-reward setting, is unavoidable.