mlsanityData to 7 Oct 2026

e-COP : Episodic Constrained Optimization of Policies

arXiv 2406.09563 ↗First posted 13 Jun 2024cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

In this paper, we present the texttte-COP algorithm, the first policy optimization algorithm for constrained Reinforcement Learning (RL) in episodic (finite horizon) settings. Such formulations are applicable when there are separate sets of optimization criteria and constraints on a system's behavior. We approach this problem by first establishing a policy difference lemma for the episodic setting, which provides the theoretical foundation for the algorithm. Then, we propose to combine a set of established and novel solution ideas to yield the texttte-COP algorithm that is easy to implement and numerically stable, and provide a theoretical guarantee on optimality under certain scaling assumptions. Through extensive empirical analysis using benchmarks in the Safety Gym suite, we show that our algorithm has similar or better performance than SoTA (non-episodic) algorithms adapted for the episodic setting. The scalability of the algorithm opens the door to its application in safety-constrained Reinforcement Learning from Human Feedback for Large Language or Diffusion Models.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 40 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

25 of 40 references.

References · 25

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  6. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  7. 2017Constrained Policy Optimizationcited by 928
  8. 2019Solving Rubik's Cube with a Robot Handcited by 699
  9. 2018Reward Constrained Policy Optimizationcited by 397
  10. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  11. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  12. 2020Projection-Based Constrained Policy Optimizationcited by 195
13 more
  1. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  2. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
  3. 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
  4. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 76
  5. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 60
  6. 2020A Unifying View of Optimism in Episodic Reinforcement Learningcited by 50
  7. 2018Online convex optimization for cumulative constraintscited by 47
  8. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 30
  9. 2021Investigating the Impact of Multi-LiDAR Placement on Object Detection for Autonomous Drivingcited by 22
  10. 2020Steady State Analysis of Episodic Reinforcement Learningcited by 21
  11. 2020Detecting Rewards Deterioration in Episodic Reinforcement Learningcited by 8
  12. 2019A Convolutional Neural Network Approach Towards Self-Driving Carscited by 4
  13. 2023ACPO: A Policy Optimization Algorithm for Average MDPs with Constraintscited by 4

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Modelscited by 13
  2. 2025Addressing Finite-Horizon MDPs via Low-Rank Tensor Value Approximationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Objective Reward and Preference Optimization: Theory and Algorithms
  2. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
  3. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
  4. 2025On the Convergence and Stability of Upside-Down Reinforcement Learning, Goal-Conditioned Supervised Learning, and Online Decision Transformers
  5. 2025Learning safe, constrained policies via imitation learning: Connection to Probabilistic Inference and a Naive Algorithm
  6. 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learning
  7. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning
  8. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control