# A Safe and Data-efficient Model-based Reinforcement Learning System for HVAC Control

arXiv: https://arxiv.org/abs/2407.12195 · HTML: https://mlsanity.com/paper/2407.12195/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 16 of its 81 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 16 of 16)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.04717/) (2016)
- [Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm](https://mlsanity.com/paper/2210.07573/) (2022)
- [Building HVAC Scheduling Using Reinforcement Learning via Neural Network Based Model Approximation](https://mlsanity.com/paper/1910.05313/) (2019)
- [Multi-zone HVAC Control with Model-Based Deep Reinforcement Learning](https://mlsanity.com/paper/2302.00725/) (2023)
- [Optimizing Irrigation Efficiency using Deep Reinforcement Learning in the Field](https://mlsanity.com/paper/2304.01435/) (2023)
- [Go Beyond Black-box Policies: Rethinking the Design of Learning Agent for Interpretable and Verifiable HVAC Control](https://mlsanity.com/paper/2403.00172/) (2024)
- [Deep Reinforcement Learning for Tropical Air Free-Cooled Data Center Control](https://mlsanity.com/paper/2012.06834/) (2020)
- [Exploring Deep Reinforcement Learning for Holistic Smart Building Control](https://mlsanity.com/paper/2301.11510/) (2023)
- [Reward Bound for Behavioral Guarantee of Model-based Planning Agents](https://mlsanity.com/paper/2402.13419/) (2024)
- [Uncertainty Estimation for Safe Human-Robot Collaboration using Conservation Measures](https://mlsanity.com/paper/2209.00467/) (2022)

## Built on by (most cited first, 5 of 5)

- [Reinforcement Learning for Sustainable Energy: A Survey](https://mlsanity.com/paper/2407.18597/) (2024)
- [Counter-Dyna: Data-Efficient RL-Based HVAC Control using Counterfactual Building Models](https://mlsanity.com/paper/2605.04555/) (2026)
- [Towards VM Rescheduling Optimization Through Deep Reinforcement Learning](https://mlsanity.com/paper/2505.17359/) (2025)
- [STEMS: Spatial-Temporal Enhanced Safe Multi-Agent Coordination for Building Energy Management](https://mlsanity.com/paper/2510.14112/) (2025)
- [Vision: "What Don't I Know About User's Preferences?" How Uncertainty Estimation Improves Proactive Assistant Alignment](https://mlsanity.com/paper/2610.15948/) (2026)

## Abstract (arXiv)

Model-Based Reinforcement Learning (MBRL) has been widely studied for Heating, Ventilation, and Air Conditioning (HVAC) control in buildings. One of the critical challenges is the large amount of data required to effectively train neural networks for modeling building dynamics. This paper presents CLUE, an MBRL system for HVAC control in buildings. CLUE optimizes HVAC operations by integrating a Gaussian Process (GP) model to model building dynamics with uncertainty awareness. CLUE utilizes GP to predict state transitions as Gaussian distributions, effectively capturing prediction uncertainty and enhancing decision-making under sparse data conditions. Our approach employs a meta-kernel learning technique to efficiently set GP kernel hyperparameters using domain knowledge from diverse buildings. This drastically reduces the data requirements typically associated with GP models in HVAC applications. Additionally, CLUE incorporates these uncertainty estimates into a Model Predictive Path Integral (MPPI) algorithm, enabling the selection of safe, energy-efficient control actions. This uncertainty-aware control strategy evaluates and selects action trajectories based on their predicted impact on energy consumption and human comfort, optimizing operations even under uncertain conditions. Extensive simulations in a five-zone office building demonstrate that CLUE reduces the required training data from hundreds of days to just seven while maintaining robust control performance. It reduces comfort violations by an average of 12.07% compared to existing MBRL methods, without compromising on energy efficiency.