# Rich State Observations Empower Reinforcement Learning to Surpass PID: A Drone Ball Balancing Study

arXiv: https://arxiv.org/abs/2509.21122 · HTML: https://mlsanity.com/paper/2509.21122/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 17 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 8 of 8)

- [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](https://mlsanity.com/paper/2301.04195/) (2023)
- [Real-World Humanoid Locomotion with Reinforcement Learning](https://mlsanity.com/paper/2303.03381/) (2023)
- [Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning](https://mlsanity.com/paper/2310.10943/) (2023)
- [A Learning-based Quadcopter Controller with Extreme Adaptation](https://mlsanity.com/paper/2409.12949/) (2024)
- [Robust Policy Optimization in Deep Reinforcement Learning](https://mlsanity.com/paper/2212.07536/) (2022)
- [Flying Hand: End-Effector-Centric Framework for Versatile Aerial Manipulation Teleoperation and Policy Learning](https://mlsanity.com/paper/2504.10334/) (2025)
- [Human-Aware Physical Human-Robot Collaborative Transportation and Manipulation with Multiple Aerial Robots](https://mlsanity.com/paper/2210.05894/) (2022)
- [Self-Organizing Aerial Swarm Robotics for Resilient Load Transportation : A Table-Mechanics-Inspired Approach](https://mlsanity.com/paper/2509.03563/) (2025)

## Abstract (arXiv)

This paper addresses a drone ball-balancing task, in which a drone stabilizes a ball atop a movable beam through cable-based interaction. We propose a hierarchical control framework that decouples high-level balancing policy from low-level drone control, and train a reinforcement learning (RL) policy to handle the high-level decision-making. Simulation results show that the RL policy achieves superior performance compared to carefully tuned PID controllers within the same hierarchical structure. Through systematic comparative analysis, we demonstrate that RL's advantage stems not from improved parameter tuning or inherent nonlinear mapping capabilities, but from its ability to effectively utilize richer state observations. These findings underscore the critical role of comprehensive state representation in learning-based systems and suggest that enhanced sensing could be instrumental in improving controller performance.