mlsanityData to 14 Oct 2026

Exact Action Values Are Not Enough: Rollout-Verified Reinforcement Fine-Tuning of a Reasoning Model for Multi-Zone VAV Control

arXiv 2607.27914 ↗First posted 30 Jul 2026cs.LG cs.SY eess.SYcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Multi-zone variable-air-volume control must balance thermal comfort, indoor air quality, and electricity use across several continuous actuators. Model predictive control and reinforcement learning are widely studied, but deployment typically requires building-specific modeling or training, limiting scalability. We first test whether a frontier reasoning model (an LLM trained to use additional inference-time computation) can achieve competitive VAV control from text without building-specific training. With that capability established, we then test whether TD3-guided reinforcement fine-tuning (RFT) can transfer control knowledge into a locally deployable open-weight model. Five controllers are evaluated over three summer days in a physics-based four-zone emulator. Relative to a Guideline 36-based baseline, TD3 reduced HVAC electricity by 4.5% while improving temperature and CO2 compliance. Without building-specific training, GPT-5 achieved the largest reduction (6.2%) but reduced the ventilation margin. For RFT, deterministic rollouts restore a saved state, apply one candidate, and follow TD3 to score each action. Auditing a learned critic against these rollouts exposed a failure hidden by its near-perfect across-time correlation (r=0.9998): within-state ranking was unreliable; the critic selected the rollout-best candidate in only 5 of 10 states. Even with the rollout verifier, 200 RFT steps produced no sustained improvement in sampled-action return; the open-weight controller used more electricity than the baseline before and after training, and its five-minute predictions remained worse than persistence. GPT-5 predicted transitions far better. Exact rollout scores rank sampled actions but reveal neither next-state effects nor an improvement direction. The unchanged transition errors motivate transition-focused supervised fine-tuning before value-based RFT.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 50 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

25 of 50 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Verifier-Based Reinforcement Fine-Tuning of Reasoning Models for Thermal Energy Storage Control14 Jul
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification9 Mar
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan

Other references · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  8. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  10. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  11. 2025Reasoning Models Don't Always Say What They Thinkcited by 415
  12. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
9 more
  1. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 124
  2. 2019Safe Contextual Bayesian Optimization for Sustainable Room Temperature PID Control Tuningcited by 24
  3. 2024Making Large Language Models into World Models with Precondition and Effect Knowledgecited by 22
  4. 2023Pre-Trained Large Language Models for Industrial Controlcited by 16
  5. 2024Go Beyond Black-box Policies: Rethinking the Design of Learning Agent for Interpretable and Verifiable HVAC Controlcited by 5
  6. 2024Large Language Model-Based Interpretable Machine Learning Control in Building Energy Systemscited by 4
  7. 2025On-line Policy Improvement using Monte-Carlo Searchcited by 3
  8. 2026ThermoLLM: Thermodynamics-Aware HVAC Control with Spatial-Semantic Knowledge Graphcited by 2
  9. 2026Hierarchical Control Framework Integrating LLMs with RL for Decarbonized HVAC Operationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Safe Deep Reinforcement Learning for Energy-Efficient HVAC Control in Multi-Zone Residential Buildings
  2. 2026Generalizing HVAC Control With Domain Randomized Reinforcement Learning
  3. 2024HVAC-DPT: A Decision Pretrained Transformer for HVAC Control
  4. 2024A Safe and Data-efficient Model-based Reinforcement Learning System for HVAC Control
  5. 2025Comparative Field Deployment of Reinforcement Learning and Model Predictive Control for Residential HVAC
  6. 2025Continual Reinforcement Learning for HVAC Systems Control: Integrating Hypernetworks and Transfer Learning
  7. 2025Lessons learned from field demonstrations of model predictive control and reinforcement learning for residential and commercial HVAC: A review
  8. 2026Counter-Dyna: Data-Efficient RL-Based HVAC Control using Counterfactual Building Models