mlsanityData to 7 Oct 2026

DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learning

arXiv 2102.11492 ↗First posted 23 Feb 2021cs.LG cs.AI cs.SYcited by 47

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Optimizing the combustion efficiency of a thermal power generating unit (TPGU) is a highly challenging and critical task in the energy industry. We develop a new data-driven AI system, namely DeepThermal, to optimize the combustion control strategy for TPGUs. At its core, is a new model-based offline reinforcement learning (RL) framework, called MORE, which leverages historical operational data of a TGPU to solve a highly complex constrained Markov decision process problem via purely offline training. In DeepThermal, we first learn a data-driven combustion process simulator from the offline dataset. The RL agent of MORE is then trained by combining real historical data as well as carefully filtered and processed simulation data through a novel restrictive exploration scheme. DeepThermal has been successfully deployed in four large coal-fired thermal power plants in China. Real-world experiments show that DeepThermal effectively improves the combustion efficiency of TPGUs. We also report the superior performance of MORE by comparing with the state-of-the-art algorithms on the standard offline RL benchmarks.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 20 of its 41 references that are in the map. First posted on arXiv 23 Feb 2021.

Builds on

20 of 41 references.

References · 20

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  12. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
8 more
  1. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  2. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  4. 2018Reward Constrained Policy Optimizationcited by 397
  5. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  6. 2020Model-Based Offline Planningcited by 126
  7. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
  8. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 75

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 10; 2023: 12; 2024: 8; 2025: 9; 2026: 3.

47 papers.

Sort:

By area:

  1. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  2. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
  3. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  4. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
  5. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
  6. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
  7. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 59
  8. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  9. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
  10. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  11. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  12. 2021Model-Based Offline Planning with Trajectory Pruningcited by 30
35 more
  1. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  2. 2022Sustainable Online Reinforcement Learning for Auto-biddingcited by 28
  3. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  4. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 20
  5. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
  6. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  7. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
  8. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
  9. 2022Semi-analytical Industrial Cooling System Model for Reinforcement Learningcited by 9
  10. 2024xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editingcited by 9
  11. 2023Machine Learning for Urban Air Quality Analytics: A Surveycited by 8
  12. 2023Offline Multi-Agent Reinforcement Learning with Coupled Value Factorizationcited by 7
  13. 2024TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agentscited by 7
  14. 2023Offline Reinforcement Learning with Imbalanced Datasetscited by 6
  15. 2022Reinforcement Learning in Practice: Opportunities and Challengescited by 5
  16. 2024A Policy Gradient Primal-Dual Algorithm for Constrained MDPs with Uniform PAC Guaranteescited by 5
  17. 2023Regularization and Variance-Weighted Regression Achieves Minimax Optimality in Linear MDPs: Theory and Practicecited by 4
  18. 2025Data Center Cooling System Optimization Using Offline Reinforcement Learningcited by 4
  19. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  20. 2023Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentationscited by 2
  21. 2023ROMO: Retrieval-enhanced Offline Model-based Optimizationcited by 1
  22. 2023A Fully Data-Driven Approach for Realistic Traffic Signal Control Using Offline Reinforcement Learningcited by 1
  23. 2024SwiftRL: Towards Efficient Reinforcement Learning on Real Processing-In-Memory Systemscited by 1
  24. 2024Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbationscited by 1
  25. 2025Nuclear Microreactor Control with Deep Reinforcement Learningcited by 1
  26. 2025Adaptive Scaling of Policy Constraints for Offline Reinforcement Learningcited by 1
  27. 2025Boundary-to-Region Supervision for Offline Safe Reinforcement Learningcited by 1
  28. 2025Online Optimization for Offline Safe Reinforcement Learningcited by 1
  29. 2025PrivORL: Differentially Private Synthetic Dataset for Offline Reinforcement Learningcited by 1
  30. 2021Efficient Robotic Manipulation Through Offline-to-Online Reinforcement Learning and Goal-Aware State Informationcited by 0
  31. 2025Multi-Fidelity Hybrid Reinforcement Learning via Information Gain Maximizationcited by 0
  32. 2025Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Controlcited by 0
  33. 2026Beyond Static Datasets: Robust Offline Policy Optimization via Vetted Synthetic Transitionscited by 0
  34. 2026Dynamics Models for Offline Hyperparameter Selection in Real-World RLcited by 0
  35. 2026Q-learning Penalized Transformer for Safe Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Offline Reinforcement-Learning-Based Power Control for Application-Agnostic Energy Efficiency
  2. 2025Safe Deployment of Offline Reinforcement Learning via Input Convex Action Correction
  3. 2026Verifier-Based Reinforcement Fine-Tuning of Reasoning Models for Thermal Energy Storage Control
  4. 2021Offline-Online Reinforcement Learning for Energy Pricing in Office Demand Response: Lowering Energy and Data Costs
  5. 2026Knowledge Distillation for Efficient Transformer-Based Reinforcement Learning in Hardware-Constrained Energy Management Systems
  6. 2025Economic Battery Storage Dispatch with Deep Reinforcement Learning from Rule-Based Demonstrations
  7. 2025Diffusion-Modeled Reinforcement Learning for Carbon and Risk-Aware Microgrid Optimization
  8. 2026Offline Reinforcement Learning for Fluid Controls: Data-based Multi-observational Policy Extraction