mlsanityData to 7 Oct 2026

Experimental evaluation of offline reinforcement learning for HVAC control in buildings

arXiv 2408.07986 ↗First posted 15 Aug 2024cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Reinforcement learning (RL) techniques have been increasingly investigated for dynamic HVAC control in buildings. However, most studies focus on exploring solutions in online or off-policy scenarios without discussing in detail the implementation feasibility or effectiveness of dealing with purely offline datasets or trajectories. The lack of these works limits the real-world deployment of RL-based HVAC controllers, especially considering the abundance of historical data. To this end, this paper comprehensively evaluates the strengths and limitations of state-of-the-art offline RL algorithms by conducting analytical and numerical studies. The analysis is conducted from two perspectives: algorithms and dataset characteristics. As a prerequisite, the necessity of applying offline RL algorithms is first confirmed in two building environments. The ability of observation history modeling to reduce violations and enhance performance is subsequently studied. Next, the performance of RL-based controllers under datasets with different qualitative and quantitative conditions is investigated, including constraint satisfaction and power consumption. Finally, the sensitivity of certain hyperparameters is also evaluated. The results indicate that datasets of a certain suboptimality level and relatively small scale can be utilized to effectively train a well-performed RL-based HVAC controller. Specifically, such controllers can reduce at most 28.5% violation ratios of indoor temperatures and achieve at most 12.1% power savings compared to the baseline controller. In summary, this paper presents our well-structured investigations and new findings when applying offline reinforcement learning to building HVAC systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 67 references that are in the map. First posted on arXiv 15 Aug 2024.

Builds on

23 of 67 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Offline RL with Observation Histories: Analyzing and Improving Sample Complexity31 Oct
  2. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?12 Apr
  3. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016OpenAI Gymcited by 1,684
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2020A Review of Deep Reinforcement Learning for Smart Building Energy Managementcited by 35
  11. 2020Multi-Agent Deep Reinforcement Learning for HVAC Control in Commercial Buildingscited by 24
  12. 2017Transforming Cooling Optimization for Green Data Center via Deep Reinforcement Learningcited by 18
3 more
  1. 2018Reinforcement Learning Testbed for Power-Consumption Optimizationcited by 18
  2. 2019Building HVAC Scheduling Using Reinforcement Learning via Neural Network Based Model Approximationcited by 14
  3. 2015Experimental analysis of data-driven control for a building heating systemcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning
  2. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  3. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  4. 2021Offline-Online Reinforcement Learning for Energy Pricing in Office Demand Response: Lowering Energy and Data Costs
  5. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  6. 2020Offline Reinforcement Learning Hands-On
  7. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  8. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning