mlsanityData to 14 Oct 2026

Dynamics Models for Offline Hyperparameter Selection in Real-World RL

arXiv 2608.11349 ↗First posted 11 Aug 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

A key obstacle to deploying reinforcement learning in real-world systems is hyperparameter selection, particularly when simulators are unavailable and online experimentation is costly. Prior work has proposed calibration models trained on offline data to approximate environment dynamics and enable offline hyperparameter selection, but these methods have so far been evaluated only in simple simulated settings. In this paper, we present the first application of calibration models in a real-world industrial setting: a municipal water treatment plant. We evaluate several calibration model approaches, including a k-nearest neighbors model with a Laplacian distance metric, on high-dimensional, non-stationary sensor data for nexting prediction tasks. Our results show that these models can generate realistic long-horizon rollouts and recover meaningful hyperparameter sensitivity trends. We further examine how calibration models scale to year-long datasets, how they support the selection of fine-tuning learning rates for pre-trained agents, and how robust they are under distribution shift. Overall, our findings provide a proof of concept for using offline dynamics models to support RL deployment in real-world environments, while highlighting important practical challenges for future work.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 33 references that are in the map. First posted on arXiv 11 Aug 2026.

Builds on

18 of 33 references.

References · 18

  1. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  2. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,013
  3. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  4. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  6. 2017Deep Reinforcement Learning that Matterscited by 754
  7. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 753
  8. 2018Learning Dexterous In-Hand Manipulationcited by 687
  9. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  10. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 110
  11. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 86
  12. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
6 more
  1. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 48
  2. 2022Investigating Compounding Prediction Errors in Learned Dynamics Modelscited by 47
  3. 2016Self-Correcting Models for Model-Based Reinforcement Learningcited by 43
  4. 2022Controlling Commercial Cooling Systems Using Reinforcement Learningcited by 29
  5. 2023GVFs in the Real World: Making Predictions Online for Water Treatmentcited by 13
  6. 2014Building a Balanced k-d Tree in O(kn log n) Timecited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2020Hyperparameter Selection for Offline Reinforcement Learning
  2. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  3. 2024The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning
  4. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data
  5. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective
  6. 2025Fully Offline Reinforcement Learning
  7. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  8. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning