# Optimizing Industrial HVAC Systems with Hierarchical Reinforcement Learning

arXiv: https://arxiv.org/abs/2209.08112 · HTML: https://mlsanity.com/paper/2209.08112/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 25 references that are in the map.
- First posted on arXiv 16 Sep 2022.

## Builds on (most cited first, 11 of 11)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018)
- [Acme: A Research Framework for Distributed Reinforcement Learning](https://mlsanity.com/paper/2006.00979/) (2020)
- [A Deep Hierarchical Approach to Lifelong Learning in Minecraft](https://mlsanity.com/paper/1604.07255/) (2016)
- [Regularized Behavior Value Estimation](https://mlsanity.com/paper/2103.09575/) (2021)
- [Semi-analytical Industrial Cooling System Model for Reinforcement Learning](https://mlsanity.com/paper/2207.13131/) (2022)
- [Energy-Efficient Thermal Comfort Control in Smart Buildings via Deep Reinforcement Learning](https://mlsanity.com/paper/1901.04693/) (2019)
- [Distributed Multi-Agent Deep Reinforcement Learning Framework for Whole-building HVAC Control](https://mlsanity.com/paper/2110.13450/) (2021)

## Built on by (most cited first, 2 of 2)

- [Anytime-Competitive Reinforcement Learning with Policy Prior](https://mlsanity.com/paper/2311.01568/) (2023)
- [Identifying Culprits Through Deep Deterministic Policy Gradient Deep Learning Investigation](https://mlsanity.com/paper/2605.14774/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) techniques have been developed to optimize industrial cooling systems, offering substantial energy savings compared to traditional heuristic policies. A major challenge in industrial control involves learning behaviors that are feasible in the real world due to machinery constraints. For example, certain actions can only be executed every few hours while other actions can be taken more frequently. Without extensive reward engineering and experimentation, an RL agent may not learn realistic operation of machinery. To address this, we use hierarchical reinforcement learning with multiple agents that control subsets of actions according to their operation time scales. Our hierarchical approach achieves energy savings over existing baselines while maintaining constraints such as operating chillers within safe bounds in a simulated HVAC control environment.