# GVFs in the Real World: Making Predictions Online for Water Treatment

arXiv: https://arxiv.org/abs/2312.01624 · HTML: https://mlsanity.com/paper/2312.01624/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 4 Dec 2023.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions](https://mlsanity.com/paper/1909.00590/) (2019)
- [Off-policy evaluation for slate recommendation](https://mlsanity.com/paper/1605.04812/) (2016)
- [Controlling Commercial Cooling Systems Using Reinforcement Learning](https://mlsanity.com/paper/2211.07357/) (2022)
- [MuZero with Self-competition for Rate Control in VP9 Video Compression](https://mlsanity.com/paper/2202.06626/) (2022)
- [A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning](https://mlsanity.com/paper/2104.13844/) (2021)
- [From Eye-blinks to State Construction: Diagnostic Benchmarks for Online Representation Learning](https://mlsanity.com/paper/2011.04590/) (2020)
- [Learning predictive representations in autonomous driving to improve deep reinforcement learning](https://mlsanity.com/paper/2006.15110/) (2020)
- [Offline Learning of Counterfactual Predictions for Real-World Robotic Reinforcement Learning](https://mlsanity.com/paper/2011.05857/) (2020)
- [Agent-State Construction with Auxiliary Inputs](https://mlsanity.com/paper/2211.07805/) (2022)

## Built on by (most cited first, 13 of 13)

- [Learning Continually by Spectral Regularization](https://mlsanity.com/paper/2406.06811/) (2024)
- [Streaming Deep Reinforcement Learning Finally Works](https://mlsanity.com/paper/2410.14606/) (2024)
- [Position: Lifetime tuning is incompatible with continual reinforcement learning](https://mlsanity.com/paper/2404.02113/) (2024)
- [Machine learning for industrial sensing and control: A survey and practical perspective](https://mlsanity.com/paper/2401.13836/) (2024)
- [A Pontryagin Perspective on Reinforcement Learning](https://mlsanity.com/paper/2405.18100/) (2024)
- [Reinitializing weights vs units for maintaining plasticity in neural networks](https://mlsanity.com/paper/2508.00212/) (2025)
- [The Cell Must Go On: Agar.io for Continual Reinforcement Learning](https://mlsanity.com/paper/2505.18347/) (2025)
- [Evaluating Feature Dependent Noise in Preference-based Reinforcement Learning](https://mlsanity.com/paper/2601.01904/) (2026)
- [Generalization in Monitored Markov Decision Processes (Mon-MDPs)](https://mlsanity.com/paper/2505.08988/) (2025)
- [Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learning](https://mlsanity.com/paper/2602.12375/) (2026)
- [Forager: a lightweight testbed for continual learning with partial observability in RL](https://mlsanity.com/paper/2605.01131/) (2026)
- [Foundations of Reinforcement Learning and Control:Connections and New Perspectives](https://mlsanity.com/paper/2608.02433/) (2026)
- [Dynamics Models for Offline Hyperparameter Selection in Real-World RL](https://mlsanity.com/paper/2608.11349/) (2026)

## Abstract (arXiv)

In this paper we investigate the use of reinforcement-learning based prediction approaches for a real drinking-water treatment plant. Developing such a prediction system is a critical step on the path to optimizing and automating water treatment. Before that, there are many questions to answer about the predictability of the data, suitable neural network architectures, how to overcome partial observability and more. We first describe this dataset, and highlight challenges with seasonality, nonstationarity, partial observability, and heterogeneity across sensors and operation modes of the plant. We then describe General Value Function (GVF) predictions -- discounted cumulative sums of observations -- and highlight why they might be preferable to classical n-step predictions common in time series prediction. We discuss how to use offline data to appropriately pre-train our temporal difference learning (TD) agents that learn these GVF predictions, including how to select hyperparameters for online fine-tuning in deployment. We find that the TD-prediction agent obtains an overall lower normalized mean-squared error than the n-step prediction agent. Finally, we show the importance of learning in deployment, by comparing a TD agent trained purely offline with no online updating to a TD agent that learns online. This final result is one of the first to motivate the importance of adapting predictions in real-time, for non-stationary high-volume systems in the real world.