mlsanityData to 14 Oct 2026

The Crucial Role of Problem Formulation in Real-World Reinforcement Learning

arXiv 2503.20442 ↗First posted 26 Mar 2025eess.SY cs.LG cs.SYcited by 6

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Reinforcement Learning (RL) offers promising solutions for control tasks in industrial cyber-physical systems (ICPSs), yet its real-world adoption remains limited. This paper demonstrates how seemingly small but well-designed modifications to the RL problem formulation can substantially improve performance, stability, and sample efficiency. We identify and investigate key elements of RL problem formulation and show that these enhance both learning speed and final policy quality. Our experiments use a one-degree-of-freedom (1-DoF) helicopter testbed, the Quanser Aero~2, which features non-linear dynamics representative of many industrial settings. In simulation, the proposed problem design principles yield more reliable and efficient training, and we further validate these results by training the agent directly on physical hardware. The encouraging real-world outcomes highlight the potential of RL for ICPS, especially when careful attention is paid to the design principles of problem formulation. Overall, our study underscores the crucial role of thoughtful problem formulation in bridging the gap between RL research and the demands of real-world industrial systems.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 7 of its 34 references that are in the map. First posted on arXiv 26 Mar 2025.

Builds on

7 of 34 references.

References · 7

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Controlcited by 53
  3. 2020Action Space Shaping in Deep Reinforcement Learningcited by 46
  4. 2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 38
  5. 2017Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learningcited by 27
  6. 2024Python-Based Reinforcement Learning on Simulink Modelscited by 9
  7. 2024Comparison of Model Predictive Control and Proximal Policy Optimization for a 1-DOF Helicopter Systemcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Multi-Objective Reinforcement Learning for Energy-Efficient Industrial Controlcited by 2
  2. 2026Impact of Markov Decision Process Design on Sim-to-Real Reinforcement Learningcited by 0
  3. 2026The Challenges of Using Reinforcement Learning for Controlling Industrial Energy Systemscited by 0
  4. 2026Anticipatory Reinforcement Learning for Trajectory Trackingcited by 0
  5. 2026Sample-Efficient Pareto Front Modeling for Energy-Aware Reinforcement Learning Using Bayesian Optimizationcited by 0
  6. 2026Safe Reinforcement Learning using Ideas from Model Predictive Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning in Operational Research: A Technical Review and Practical Roadmap
  2. 2025Logic-informed reinforcement learning for cross-domain optimization of large-scale cyber-physical systems
  3. 2025A Survey of Reinforcement Learning for Optimization in Automation
  4. 2025Evaluating Reinforcement Learning Safety and Trustworthiness in Cyber-Physical Systems
  5. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions
  6. 2026Integrating Physics-Informed Neural Networks for Safe Reinforcement Learning in a 1-DoF Helicopter System
  7. 2025Guidelines for Applying RL and MARL in Cybersecurity Applications
  8. 2025Mission-Aligned Learning-Informed Control of Autonomous Systems: Formulation and Foundations