mlsanityData to 14 Oct 2026

Counter-Dyna: Data-Efficient RL-Based HVAC Control using Counterfactual Building Models

arXiv 2605.04555 ↗First posted 6 May 2026cs.LG cs.SY eess.SYcited by 2

On the map: Deep reinforcement learning methods → RL for energy, networks and operations

Abstract (arXiv)

Model-based reinforcement learning (MBRL) offers a promising approach for data-efficient energy management in buildings, combining the strengths of predictive modeling and reinforcement learning. While previous MBRL methods applied to HVAC control have reduced training data requirements, they still require several months of interaction with the building to learn a satisfactory control policy. A key reason is that existing surrogate models attempt to predict the entire state-space, including weather and electricity prices that are unaffected by control actions, or completely ignore these variables. Addressing these issues, we propose Counter-Dyna, a method that enhances the data-efficiency of Dyna, an MBRL method. We create data-efficient counterfactual surrogate models (CSM) by leveraging invariances in the state-space. Using a CSM in Dyna speeds up RL training measured in environment interaction data compared to previous results. In comparison with previous state-of-the-art that used 6-12 months of environment interactions, our method needs only 5 weeks. We evaluate our method in a large simulation study using the literature standard BOPTEST framework and proximal policy algorithm (PPO) as the RL algorithm. Our results show cost-saving potentials of 5.3% to 17.0% in a hypothetical deployment scenario. Our work is a significant step towards making real-world deployment of RL algorithms in HVAC control practically viable.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods. Builds on 9 of its 33 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

9 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024A Safe and Data-efficient Model-based Reinforcement Learning System for HVAC Control16 Jul

Other references · 8

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  5. 2020A Review of Deep Reinforcement Learning for Smart Building Energy Managementcited by 38
  6. 2025GenTL: A General Transfer Learning Model for Building Thermal Dynamicscited by 9
  7. 2025Adapting to Change: A Comparison of Continual and Transfer Learning for Modeling Building Thermal Dynamics under Concept Driftscited by 7
  8. 2021Development of a Soft Actor Critic Deep Reinforcement Learning Approach for Harnessing Energy Flexibility in a Large Office Buildingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026BuilDyn: Excitation-Driven Data Generation for Building Thermal Dynamics Modeling and Controlcited by 1
  2. 2026Very Exciting: Zero-Shot Model Predictive Control of Buildings via Excitation-Based Generalized Transfer Learning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Experimental evaluation of offline reinforcement learning for HVAC control in buildings
  2. 2025Towards Machine Learning-based Model Predictive Control for HVAC Control in Multi-Context Buildings at Scale via Ensemble Learning
  3. 2026Hierarchical Control Framework Integrating LLMs with RL for Decarbonized HVAC Operation
  4. 2025Continual Reinforcement Learning for HVAC Systems Control: Integrating Hypernetworks and Transfer Learning
  5. 2024HVAC-DPT: A Decision Pretrained Transformer for HVAC Control
  6. 2025Reinforcement Learning (RL) Meets Urban Climate Modeling: Investigating the Efficacy and Impacts of RL-Based HVAC Control
  7. 2024Go Beyond Black-box Policies: Rethinking the Design of Learning Agent for Interpretable and Verifiable HVAC Control
  8. 2024An experimental evaluation of Deep Reinforcement Learning algorithms for HVAC control