# A Lyapunov Drift-Plus-Penalty Method Tailored for Reinforcement Learning with Queue Stability

arXiv: https://arxiv.org/abs/2506.04291 · HTML: https://mlsanity.com/paper/2506.04291/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 9 of its 31 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 9 of 9)

- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Exploration-Exploitation in Constrained MDPs](https://mlsanity.com/paper/2003.02189/) (2020)
- [Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods](https://mlsanity.com/paper/2404.00282/) (2024)
- [Deep Reinforcement Learning for Stochastic Computation Offloading in Digital Twin Networks](https://mlsanity.com/paper/2011.08430/) (2020)
- [Lyapunov-guided Deep Reinforcement Learning for Stable Online Computation Offloading in Mobile-Edge Computing Networks](https://mlsanity.com/paper/2010.01370/) (2020)
- [Energy-Efficient Federated Edge Learning with Streaming Data: A Lyapunov Optimization Approach](https://mlsanity.com/paper/2405.12046/) (2024)
- [Achieving Instance-dependent Sample Complexity for Constrained Markov Decision Process](https://mlsanity.com/paper/2402.16324/) (2024)
- [On the Convergence Time of Dual Subgradient Methods for Strongly Convex Programs](https://mlsanity.com/paper/1503.06235/) (2015)
- [Energy-Latency Aware Intelligent Reflecting Surface Aided Multi-cell Mobile Edge Computing](https://mlsanity.com/paper/2305.03556/) (2023)

## Built on by (most cited first, 2 of 2)

- [Theoretical Foundations of Continual Learning via Drift-Plus-Penalty](https://mlsanity.com/paper/2606.08452/) (2026)
- [Federated Continual Learning as a Distributed Drift-Plus-Penalty Control Problem](https://mlsanity.com/paper/2608.21539/) (2026)

## Abstract (arXiv)

With the proliferation of Internet of Things (IoT) devices, the demand for addressing complex optimization challenges has intensified. The Lyapunov Drift-Plus-Penalty algorithm is a widely adopted approach for ensuring queue stability, and some research has preliminarily explored its integration with reinforcement learning (RL). In this paper, we investigate the adaptation of the Lyapunov Drift-Plus-Penalty algorithm for RL applications, deriving an effective method for combining Lyapunov Drift-Plus-Penalty with RL under a set of common and reasonable conditions through rigorous theoretical analysis. Unlike existing approaches that directly merge the two frameworks, our proposed algorithm, termed Lyapunov drift-plus-penalty method tailored for reinforcement learning with queue stability (LDPTRLQ) algorithm, offers theoretical superiority by effectively balancing the greedy optimization of Lyapunov Drift-Plus-Penalty with the long-term perspective of RL. Simulation results for multiple problems demonstrate that LDPTRLQ outperforms the baseline methods using the Lyapunov drift-plus-penalty method and RL, corroborating the validity of our theoretical derivations. The results also demonstrate that our proposed algorithm outperforms other benchmarks in terms of compatibility and stability.