# Controlling Commercial Cooling Systems Using Reinforcement Learning

arXiv: https://arxiv.org/abs/2211.07357 · HTML: https://mlsanity.com/paper/2211.07357/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 11 Nov 2022.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble](https://mlsanity.com/paper/2110.01548/) (2021)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion](https://mlsanity.com/paper/1807.01675/) (2018)
- [Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters](https://mlsanity.com/paper/2205.13703/) (2022)
- [A Review of Deep Reinforcement Learning for Smart Building Energy Management](https://mlsanity.com/paper/2008.05074/) (2020)
- [Regularized Behavior Value Estimation](https://mlsanity.com/paper/2103.09575/) (2021)
- [Semi-analytical Industrial Cooling System Model for Reinforcement Learning](https://mlsanity.com/paper/2207.13131/) (2022)
- [TD or not TD: Analyzing the Role of Temporal Differencing in Deep Reinforcement Learning](https://mlsanity.com/paper/1806.01175/) (2018)

## Built on by (most cited first, 29 of 29)

- [Loss of Plasticity in Continual Deep Reinforcement Learning](https://mlsanity.com/paper/2303.07507/) (2023)
- [TorchRL: A data-driven decision-making library for PyTorch](https://mlsanity.com/paper/2306.00577/) (2023)
- [ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs](https://mlsanity.com/paper/2302.01275/) (2023)
- [MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking](https://mlsanity.com/paper/2501.13011/) (2025)
- [GVFs in the Real World: Making Predictions Online for Water Treatment](https://mlsanity.com/paper/2312.01624/) (2023)
- [Train Hard, Fight Easy: Robust Meta Reinforcement Learning](https://mlsanity.com/paper/2301.11147/) (2023)
- [Position: Lifetime tuning is incompatible with continual reinforcement learning](https://mlsanity.com/paper/2404.02113/) (2024)
- [A Review of Safe Reinforcement Learning Methods for Modern Power Systems](https://mlsanity.com/paper/2407.00304/) (2024)
- [Towards practical reinforcement learning for tokamak magnetic control](https://mlsanity.com/paper/2307.11546/) (2023)
- [CoDreamer: Communication-Based Decentralised World Models](https://mlsanity.com/paper/2406.13600/) (2024)
- [Stealthy Imitation: Reward-guided Environment-free Policy Stealing](https://mlsanity.com/paper/2405.07004/) (2024)
- [State and Action Factorization in Power Grids](https://mlsanity.com/paper/2409.04467/) (2024)
- [Lessons learned from field demonstrations of model predictive control and reinforcement learning for residential and commercial HVAC: A review](https://mlsanity.com/paper/2503.05022/) (2025)
- [AI Finds A Way](https://mlsanity.com/paper/2608.23875/) (2026)
- [Transfer Learning on Transformers for Building Energy Consumption Forecasting -- A Comparative Study](https://mlsanity.com/paper/2410.14107/) (2024)
- [Reinforcement Learning Based Power Grid Day-Ahead Planning and AI-Assisted Control](https://mlsanity.com/paper/2302.07654/) (2023)
- [Anytime-Competitive Reinforcement Learning with Policy Prior](https://mlsanity.com/paper/2311.01568/) (2023)
- [Black Box Meta-Learning Intrinsic Rewards](https://mlsanity.com/paper/2407.21546/) (2024)
- [Nuclear Microreactor Control with Deep Reinforcement Learning](https://mlsanity.com/paper/2504.00156/) (2025)
- [Control of Renewable Energy Communities using AI and Real-World Data](https://mlsanity.com/paper/2505.17321/) (2025)
- [Action Shapley: A Training Data Selection Metric for World Model in Reinforcement Learning](https://mlsanity.com/paper/2601.10905/) (2026)
- [Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy](https://mlsanity.com/paper/2606.28433/) (2026)
- [Learning Visual Tracking and Reaching with Deep Reinforcement Learning on a UR10e Robotic Arm](https://mlsanity.com/paper/2308.14652/) (2023)
- [Interactive Design-of-Experiments: Optimizing a Cooling System](https://mlsanity.com/paper/2408.12607/) (2024)
- [Effective Reward Specification in Deep Reinforcement Learning](https://mlsanity.com/paper/2412.07177/) (2024)
- [Comparative Field Deployment of Reinforcement Learning and Model Predictive Control for Residential HVAC](https://mlsanity.com/paper/2510.01475/) (2025)
- [A Receding Horizon Reinforcement Learning Framework for Campus Chiller Energy Management - A case study from an Australian University](https://mlsanity.com/paper/2511.14160/) (2025)
- [Dynamics Models for Offline Hyperparameter Selection in Real-World RL](https://mlsanity.com/paper/2608.11349/) (2026)
- [Admissable: Training Reinforcement Learning Agents against Adversarial Missingness](https://mlsanity.com/paper/2609.15297/) (2026)

## Abstract (arXiv)

This paper is a technical overview of DeepMind and Google's recent work on reinforcement learning for controlling commercial cooling systems. Building on expertise that began with cooling Google's data centers more efficiently, we recently conducted live experiments on two real-world facilities in partnership with Trane Technologies, a building management system provider. These live experiments had a variety of challenges in areas such as evaluation, learning from offline data, and constraint satisfaction. Our paper describes these challenges in the hope that awareness of them will benefit future applied RL work. We also describe the way we adapted our RL system to deal with these challenges, resulting in energy savings of approximately 9% and 13% respectively at the two live experiment sites.