mlsanityData to 14 Oct 2026

Anytime-Competitive Reinforcement Learning with Policy Prior

arXiv 2311.01568 ↗First posted 2 Nov 2023cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

This paper studies the problem of Anytime-Competitive Markov Decision Process (A-CMDP). Existing works on Constrained Markov Decision Processes (CMDPs) aim to optimize the expected reward while constraining the expected cost over random dynamics, but the cost in a specific episode can still be unsatisfactorily high. In contrast, the goal of A-CMDP is to optimize the expected reward while guaranteeing a bounded cost in each round of any episode against a policy prior. We propose a new algorithm, called Anytime-Competitive Reinforcement Learning (ACRL), which provably guarantees the anytime cost constraints. The regret analysis shows the policy asymptotically matches the optimal reward achievable under the anytime competitive constraints. Experiments on the application of carbon-intelligent computing verify the reward performance and cost constraint guarantee of ACRL.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 51 of its 72 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

51 of 72 references.

References · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Constrained Policy Optimizationcited by 983
  3. 2019Green AIcited by 627
  4. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 599
  5. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  6. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
  7. 2013Learning to Optimize Via Posterior Samplingcited by 256
  8. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  9. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 234
  10. 2020Projection-Based Constrained Policy Optimizationcited by 203
  11. 2021DC3: A learning method for optimization with hard constraintscited by 184
  12. 2020Exploration-Exploitation in Constrained MDPscited by 149
38 more
  1. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  2. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  3. 2022Measuring the Carbon Intensity of AI in Cloud Instancescited by 111
  4. 2020Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mappingcited by 102
  5. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 91
  6. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 69
  7. 2022A Survey on Model-based Reinforcement Learningcited by 66
  8. 2016Conservative Banditscited by 59
  9. 2021Contraction Theory for Nonlinear Stability Analysis and Learning-based Control: A Tutorial Overviewcited by 45
  10. 2021Carbon-Aware Computing for Datacenterscited by 41
  11. 2022Near-Optimal Sample Complexity Bounds for Constrained MDPscited by 37
  12. 2021Safe Reinforcement Learning with Linear Function Approximationcited by 35
  13. 2021Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimizationcited by 31
  14. 2022Provably Efficient Model-Free Constrained RL with Linear Function Approximationcited by 31
  15. 2022Controlling Commercial Cooling Systems Using Reinforcement Learningcited by 29
  16. 2021Perturbation-based Regret Analysis of Predictive Control in Linear Time Varying Systemscited by 25
  17. 2021On the Regret Analysis of Online LQR Control with Predictionscited by 22
  18. 2020Conservative Exploration in Reinforcement Learningcited by 20
  19. 2021Sublinear Regret for Learning POMDPscited by 19
  20. 2022Bounded-Regret MPC via Perturbation Analysis: Prediction Error, Constraints, and Nonlinearitycited by 16
  21. 2020Competitive Algorithms for the Online Multiple Knapsack Problem with Application to Electric Vehicle Chargingcited by 15
  22. 2021Stability Constrained Reinforcement Learning for Real-Time Voltage Controlcited by 15
  23. 2022An Adaptive Deep RL Method for Non-Stationary Environments with Piecewise Stable Contextcited by 15
  24. 2021Robustness and Consistency in Linear Quadratic Control with Untrusted Predictionscited by 14
  25. 2021Safe Adaptive Learning-based Control for Constrained Linear Quadratic Regulators with Regret Guaranteescited by 11
  26. 2021Reinforcement Learning with Almost Sure Constraintscited by 11
  27. 2023Reinforcement Learning with History-Dependent Dynamic Contextscited by 11
  28. 2021A Reduction-Based Framework for Conservative Bandits and Reinforcement Learningcited by 9
  29. 2022Semi-analytical Industrial Cooling System Model for Reinforcement Learningcited by 9
  30. 2023Online switching control with stability and regret guaranteescited by 9
  31. 2022Expert-Calibrated Learning for Online Optimization with Switching Costscited by 6
  32. 2020A One-Size-Fits-All Solution to Conservative Bandit Problemscited by 5
  33. 2022Policy Gradients for Probabilistic Constrained Reinforcement Learningcited by 5
  34. 2018Conservative Exploration using Interleavingcited by 4
  35. 2023Robustified Learning for Online Optimization with Memory Costscited by 4
  36. 2018Reverse and Forward Engineering of Local Voltage Control in Distribution Networkscited by 3
  37. 2020Asymptotically Optimal Load Balancing in Large-scale Heterogeneous Systems with Multiple Dispatcherscited by 3
  38. 2022Optimizing Industrial HVAC Systems with Hierarchical Reinforcement Learningcited by 2

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Safe Exploitative Play with Untrusted Type Beliefscited by 0