# Reinforcement Learning with Inner-loop Dynamics Estimator for Aerial Manipulation under Uncertainty

arXiv: https://arxiv.org/abs/2606.16621 · HTML: https://mlsanity.com/paper/2606.16621/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Aerial Gym Simulator: A Framework for Highly Parallelized Simulation of Aerial Robots](https://mlsanity.com/paper/2503.01471/) (2025)
- [Actuator Trajectory Planning for UAVs with Overhead Manipulator using Reinforcement Learning](https://mlsanity.com/paper/2308.12843/) (2023)
- [AERMANI-VLM: Structured Prompting and Reasoning for Aerial Manipulation with Vision Language Models](https://mlsanity.com/paper/2511.01472/) (2025)
- [Global End-Effector Pose Control of an Underactuated Aerial Manipulator via Reinforcement Learning](https://mlsanity.com/paper/2512.21085/) (2025)
- [Modular Adaptive Aerial Manipulation under Unknown Dynamic Coupling Forces](https://mlsanity.com/paper/2410.08285/) (2024)
- [Learning to Open Doors with an Aerial Manipulator](https://mlsanity.com/paper/2307.15581/) (2023)
- [Computation-Aware Learning for Stable Control with Gaussian Process](https://mlsanity.com/paper/2406.02272/) (2024)
- [AERMANI-Diffusion: Regime-Conditioned Diffusion for Dynamics Learning in Aerial Manipulators](https://mlsanity.com/paper/2512.10773/) (2025)
- [ARCADE: Adaptive Robot Control with Online Changepoint-Aware Bayesian Dynamics Learning](https://mlsanity.com/paper/2512.14331/) (2025)
- [AeroPlace-Flow: Language-Grounded Object Placement for Aerial Manipulators via Visual Foresight and Object Flow](https://mlsanity.com/paper/2603.07744/) (2026)
- [AeroGrab: A Unified Framework for Aerial Grasping in Cluttered Environments](https://mlsanity.com/paper/2603.15097/) (2026)
- [An Integrated Approach to Aerial Grasping: Combining a Bistable Gripper with Adaptive Control](https://mlsanity.com/paper/2311.00769/) (2023)
- [Robust Control of An Aerial Manipulator Based on A Variable Inertia Parameters Model](https://mlsanity.com/paper/2401.04316/) (2024)
- [Impedance and Stability Targeted Adaptation for Aerial Manipulator with Unknown Coupling Dynamics](https://mlsanity.com/paper/2504.01983/) (2025)
- [Learn Structure, Adapt on the Fly: Multi-Scale Residual Learning and Online Adaptation for Aerial Manipulators](https://mlsanity.com/paper/2603.11638/) (2026)
- [Learning Cross-Coupled and Regime Dependent Dynamics for Aerial Manipulation](https://mlsanity.com/paper/2605.14805/) (2026)
- [Physics-Aware Sparse Learning and Selective Online Adaptation for Euler-Lagrange Robot Dynamics](https://mlsanity.com/paper/2606.09640/) (2026)

## Built on by (most cited first, 1 of 1)

- [UAVs Meet Embodied Intelligence: Bridging Human Intents and Flying Dynamics Via Harnessing Physical-Digital AI Agents](https://mlsanity.com/paper/2609.18326/) (2026)

## Abstract (arXiv)

Aerial manipulators enable physical interaction in hard-to-reach environments; however, the combined problem of direct whole-body aerial manipulation under rapid arm motion, payload changes, and related unknown dynamic uncertainty remains a largely unsolved problem. We present a hierarchical control framework that combines Reinforcement Learning (RL) with an inner-loop dynamics estimator to address this problem. The RL outer loop maps desired 6-degrees-of-freedom (DOF) end-effector targets to coordinated whole-body commands, enabling direct task-driven control without relying on a fully accurate coupled dynamic model in the policy layer. An inner loop then tracks these commands while compensating for transient inertial shifts and uncertainty during execution via a dynamics estimator scheme without requiring system model knowledge. We validate the proposed approach on a custom quadrotor equipped with a 3-DoF manipulator through hardware experiments under varying payload conditions. Compared with RL+PID and RL+INDI+PID baselines, the proposed method reduces end-effector tracking error and improves task success rate across the tested hardware conditions. These results show that combining learned whole-body coordination with estimator-based low-level compensation improves the precision and robustness of aerial manipulation under changing operating conditions.