# Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimization

arXiv: https://arxiv.org/abs/2105.08881 · HTML: https://mlsanity.com/paper/2105.08881/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 157 references that are in the map.
- First posted on arXiv 19 May 2021.

## Builds on (most cited first, 30 of 33)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Tackling Climate Change with Machine Learning](https://mlsanity.com/paper/1906.05433/) (2019)
- [Projection-Based Constrained Policy Optimization](https://mlsanity.com/paper/2010.03152/) (2020)
- [SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver](https://mlsanity.com/paper/1905.12149/) (2019)
- [Neural Lyapunov Control](https://mlsanity.com/paper/2005.00611/) (2020)
- [Predicting AC Optimal Power Flows: Combining Deep Learning and Lagrangian Dual Methods](https://mlsanity.com/paper/1909.10461/) (2019)
- [Deep Declarative Networks: A New Hope](https://mlsanity.com/paper/1909.04866/) (2019)
- [Learning Optimal Solutions for Extremely Fast AC Optimal Power Flow](https://mlsanity.com/paper/1910.01213/) (2019)
- [Safe Exploration in Finite Markov Decision Processes with Gaussian Processes](https://mlsanity.com/paper/1606.04753/) (2016)
- [OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World](https://mlsanity.com/paper/1709.07643/) (2017)
- [Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergence](https://mlsanity.com/paper/1910.09496/) (2019)
- [Enforcing robust control guarantees within neural network policies](https://mlsanity.com/paper/2011.08105/) (2020)
- [Verifiably Safe Exploration for End-to-End Reinforcement Learning](https://mlsanity.com/paper/2007.01223/) (2020)
- [What game are we playing? End-to-end learning in normal and extensive form games](https://mlsanity.com/paper/1805.02777/) (2018)
- [Differentiable Submodular Maximization](https://mlsanity.com/paper/1803.01785/) (2018)
- [Verifiably Safe Off-Model Reinforcement Learning](https://mlsanity.com/paper/1902.05632/) (2019)
- [High-Fidelity Machine Learning Approximations of Large-Scale Optimal Power Flow](https://mlsanity.com/paper/2006.16356/) (2020)
- [Solving Online Threat Screening Games using Constrained Action Space Reinforcement Learning](https://mlsanity.com/paper/1911.08799/) (2019)
- [Designing reactive power control rules for smart inverters using support vector machines](https://mlsanity.com/paper/1903.01016/) (2019)
- [H_∞ Model-free Reinforcement Learning with Robust Stability Guarantee](https://mlsanity.com/paper/1911.02875/) (2019)
- [COHORT: Coordination of Heterogeneous Thermostatically Controlled Loads for Demand Flexibility](https://mlsanity.com/paper/2010.03659/) (2020)

## Built on by (most cited first, 30 of 31)

- [A Survey of Contextual Optimization Methods for Decision Making under Uncertainty](https://mlsanity.com/paper/2306.10374/) (2023)
- [Self-Supervised Primal-Dual Learning for Constrained Optimization](https://mlsanity.com/paper/2208.09046/) (2022)
- [Safe Pontryagin Differentiable Programming](https://mlsanity.com/paper/2105.14937/) (2021)
- [Application-Driven Innovation in Machine Learning](https://mlsanity.com/paper/2403.17381/) (2024)
- [Conditional Synthetic Data Generation for Robust Machine Learning Applications with Limited Pandemic Data](https://mlsanity.com/paper/2109.06486/) (2021)
- [Softened Symbol Grounding for Neuro-symbolic Systems](https://mlsanity.com/paper/2403.00323/) (2024)
- [Differentiating Through Integer Linear Programs with Quadratic Regularization and Davis-Yin Splitting](https://mlsanity.com/paper/2301.13395/) (2023)
- [Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control with Action Constraints](https://mlsanity.com/paper/2304.08743/) (2023)
- [Enforcing Hard Linear Constraints in Deep Learning Models with Decision Rules](https://mlsanity.com/paper/2505.13858/) (2025)
- [Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs](https://mlsanity.com/paper/2503.12932/) (2025)
- [The Smart Buildings Control Suite: A Diverse Open Source Benchmark to Evaluate and Scale HVAC Control Policies for Sustainability](https://mlsanity.com/paper/2410.03756/) (2024)
- [Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?](https://mlsanity.com/paper/2509.12833/) (2025)
- [CDCGen: Cross-Domain Conditional Generation via Normalizing Flows and Adversarial Training](https://mlsanity.com/paper/2108.11368/) (2021)
- [Near-optimal Deep Reinforcement Learning Policies from Data for Zone Temperature Control](https://mlsanity.com/paper/2203.05434/) (2022)
- [CommonPower: A Framework for Safe Data-Driven Smart Grid Control](https://mlsanity.com/paper/2406.03231/) (2024)
- [You Shall Pass: Dealing with the Zero-Gradient Problem in Predict and Optimize for Convex Optimization](https://mlsanity.com/paper/2307.16304/) (2023)
- [Improving Feasibility via Fast Autoencoder-Based Projections](https://mlsanity.com/paper/2604.03489/) (2026)
- [Self-supervised Equality Embedded Deep Lagrange Dual for Approximate Constrained Optimization](https://mlsanity.com/paper/2306.06674/) (2023)
- [DFWLayer: Differentiable Frank-Wolfe Optimization Layer](https://mlsanity.com/paper/2308.10806/) (2023)
- [E2E-AT: A Unified Framework for Tackling Uncertainty in Task-aware End-to-end Learning](https://mlsanity.com/paper/2312.10587/) (2023)
- [Conditional Synthetic Data Generation for Personal Thermal Comfort Models](https://mlsanity.com/paper/2203.05242/) (2022)
- [Multi-market Energy Optimization with Renewables via Reinforcement Learning](https://mlsanity.com/paper/2306.08147/) (2023)
- [Anytime-Competitive Reinforcement Learning with Policy Prior](https://mlsanity.com/paper/2311.01568/) (2023)
- [Two-Stage Predict+Optimize for Mixed Integer Linear Programs with Unknown Parameters in Constraints](https://mlsanity.com/paper/2311.08022/) (2023)
- [Control Policy Correction Framework for Reinforcement Learning-based Energy Arbitrage Strategies](https://mlsanity.com/paper/2404.18821/) (2024)
- [Leveraging Analytic Gradients in Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2506.01665/) (2025)
- [T-SKM-Net: Trainable Neural Network Framework for Linear Constraint Satisfaction via Sampling Kaczmarz-Motzkin Method](https://mlsanity.com/paper/2512.10461/) (2025)
- [Solving Max-Cut to Global Optimality via Feasibility-Preserving Graph Neural Networks](https://mlsanity.com/paper/2605.07113/) (2026)
- [Computationally Efficient Reinforcement Learning: Targeted Exploration leveraging Simple Rules](https://mlsanity.com/paper/2211.16691/) (2022)
- [Efficient Reinforcement Learning Through Trajectory Generation](https://mlsanity.com/paper/2211.17249/) (2022)

## Abstract (arXiv)

While reinforcement learning (RL) is gaining popularity in energy systems control, its real-world applications are limited due to the fact that the actions from learned policies may not satisfy functional requirements or be feasible for the underlying physical system. In this work, we propose PROjected Feasibility (PROF), a method to enforce convex operational constraints within neural policies. Specifically, we incorporate a differentiable projection layer within a neural network-based policy to enforce that all learned actions are feasible. We then update the policy end-to-end by propagating gradients through this differentiable projection layer, making the policy cognizant of the operational constraints. We demonstrate our method on two applications: energy-efficient building operation and inverter control. In the building operation setting, we show that PROF maintains thermal comfort requirements while improving energy efficiency by 4% over state-of-the-art methods. In the inverter control setting, PROF perfectly satisfies voltage constraints on the IEEE 37-bus feeder system, as it learns to curtail as little renewable energy as possible within its safety set.