mlsanityData to 7 Oct 2026

GVFs in the Real World: Making Predictions Online for Water Treatment

arXiv 2312.01624 ↗First posted 4 Dec 2023cs.LG cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

In this paper we investigate the use of reinforcement-learning based prediction approaches for a real drinking-water treatment plant. Developing such a prediction system is a critical step on the path to optimizing and automating water treatment. Before that, there are many questions to answer about the predictability of the data, suitable neural network architectures, how to overcome partial observability and more. We first describe this dataset, and highlight challenges with seasonality, nonstationarity, partial observability, and heterogeneity across sensors and operation modes of the plant. We then describe General Value Function (GVF) predictions -- discounted cumulative sums of observations -- and highlight why they might be preferable to classical n-step predictions common in time series prediction. We discuss how to use offline data to appropriately pre-train our temporal difference learning (TD) agents that learn these GVF predictions, including how to select hyperparameters for online fine-tuning in deployment. We find that the TD-prediction agent obtains an overall lower normalized mean-squared error than the n-step prediction agent. Finally, we show the importance of learning in deployment, by comparing a TD agent trained purely offline with no online updating to a TD agent that learns online. This final result is one of the first to motivate the importance of adapting predictions in real-time, for non-stationary high-volume systems in the real world.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 17 of its 45 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

17 of 45 references.

References · 17

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  6. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  7. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  8. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  9. 2019Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directionscited by 172
  10. 2016Off-policy evaluation for slate recommendationcited by 102
  11. 2022MuZero with Self-competition for Rate Control in VP9 Video Compressioncited by 27
  12. 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
5 more
  1. 2022Controlling Commercial Cooling Systems Using Reinforcement Learningcited by 25
  2. 2020From Eye-blinks to State Construction: Diagnostic Benchmarks for Online Representation Learningcited by 13
  3. 2020Learning predictive representations in autonomous driving to improve deep reinforcement learningcited by 6
  4. 2020Offline Learning of Counterfactual Predictions for Real-World Robotic Reinforcement Learningcited by 3
  5. 2022Agent-State Construction with Auxiliary Inputscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 3; 2026: 5.

13 papers.

Sort:

By area:

  1. 2024Learning Continually by Spectral Regularizationcited by 34
  2. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  3. 2024Position: Lifetime tuning is incompatible with continual reinforcement learningcited by 9
  4. 2024A Pontryagin Perspective on Reinforcement Learningcited by 7
  5. 2024Machine learning for industrial sensing and control: A survey and practical perspectivecited by 6
  6. 2025Reinitializing weights vs units for maintaining plasticity in neural networkscited by 6
  7. 2025The Cell Must Go On: Agar.io for Continual Reinforcement Learningcited by 2
  8. 2025Generalization in Monitored Markov Decision Processes (Mon-MDPs)cited by 1
  9. 2026Evaluating Feature Dependent Noise in Preference-based Reinforcement Learningcited by 1
  10. 2026Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learningcited by 1
  11. 2026Forager: a lightweight testbed for continual learning with partial observability in RLcited by 0
  12. 2026Foundations of Reinforcement Learning and Control:Connections and New Perspectivescited by 0
1 more
  1. 2026Dynamics Models for Offline Hyperparameter Selection in Real-World RLcited by 0