# The Crucial Role of Problem Formulation in Real-World Reinforcement Learning

arXiv: https://arxiv.org/abs/2503.20442 · HTML: https://mlsanity.com/paper/2503.20442/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 7 of its 34 references that are in the map.
- First posted on arXiv 26 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 7 of 7)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control](https://mlsanity.com/paper/1706.06491/) (2017)
- [Action Space Shaping in Deep Reinforcement Learning](https://mlsanity.com/paper/2004.00980/) (2020)
- [Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applications](https://mlsanity.com/paper/2408.10215/) (2024)
- [Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning](https://mlsanity.com/paper/1703.03078/) (2017)
- [Python-Based Reinforcement Learning on Simulink Models](https://mlsanity.com/paper/2405.08567/) (2024)
- [Comparison of Model Predictive Control and Proximal Policy Optimization for a 1-DOF Helicopter System](https://mlsanity.com/paper/2408.15633/) (2024)

## Built on by (most cited first, 6 of 6)

- [Multi-Objective Reinforcement Learning for Energy-Efficient Industrial Control](https://mlsanity.com/paper/2505.07607/) (2025)
- [Impact of Markov Decision Process Design on Sim-to-Real Reinforcement Learning](https://mlsanity.com/paper/2603.09427/) (2026)
- [The Challenges of Using Reinforcement Learning for Controlling Industrial Energy Systems](https://mlsanity.com/paper/2605.31044/) (2026)
- [Anticipatory Reinforcement Learning for Trajectory Tracking](https://mlsanity.com/paper/2607.03132/) (2026)
- [Sample-Efficient Pareto Front Modeling for Energy-Aware Reinforcement Learning Using Bayesian Optimization](https://mlsanity.com/paper/2607.03140/) (2026)
- [Safe Reinforcement Learning using Ideas from Model Predictive Control](https://mlsanity.com/paper/2607.07252/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) offers promising solutions for control tasks in industrial cyber-physical systems (ICPSs), yet its real-world adoption remains limited. This paper demonstrates how seemingly small but well-designed modifications to the RL problem formulation can substantially improve performance, stability, and sample efficiency. We identify and investigate key elements of RL problem formulation and show that these enhance both learning speed and final policy quality. Our experiments use a one-degree-of-freedom (1-DoF) helicopter testbed, the Quanser Aero~2, which features non-linear dynamics representative of many industrial settings. In simulation, the proposed problem design principles yield more reliable and efficient training, and we further validate these results by training the agent directly on physical hardware. The encouraging real-world outcomes highlight the potential of RL for ICPS, especially when careful attention is paid to the design principles of problem formulation. Overall, our study underscores the crucial role of thoughtful problem formulation in bridging the gap between RL research and the demands of real-world industrial systems.