# Meta-Adaptive Beam Search Planning for Transformer-Based Reinforcement Learning Control of UAVs with Overhead Manipulators under Flight Disturbances

arXiv: https://arxiv.org/abs/2603.26612 · HTML: https://mlsanity.com/paper/2603.26612/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 18 of its 47 references that are in the map.
- First posted on arXiv 27 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 18 of 18)

- [CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models](https://mlsanity.com/paper/2503.22020/) (2025)
- [On Transforming Reinforcement Learning by Transformer: The Development Trajectory](https://mlsanity.com/paper/2212.14164/) (2022)
- [Critic-Guided Decision Transformer for Offline Reinforcement Learning](https://mlsanity.com/paper/2312.13716/) (2023)
- [A comprehensive survey of research towards AI-enabled unmanned aerial systems in pre-, active-, and post-wildfire management](https://mlsanity.com/paper/2401.02456/) (2024)
- [Nonlinear MPC for Collision Avoidance and Controlof UAVs With Dynamic Obstacles](https://mlsanity.com/paper/2008.00792/) (2020)
- [Transformers in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.05979/) (2023)
- [Beyond Greedy Search: Tracking by Multi-Agent Reinforcement Learning-based Beam Search](https://mlsanity.com/paper/2205.09676/) (2022)
- [Actuator Trajectory Planning for UAVs with Overhead Manipulator using Reinforcement Learning](https://mlsanity.com/paper/2308.12843/) (2023)
- [Monte Carlo Beam Search for Actor-Critic Reinforcement Learning in Continuous Control](https://mlsanity.com/paper/2505.09029/) (2025)
- [Aerial Manipulation using Model Predictive Control for Opening a Hinged Door](https://mlsanity.com/paper/2003.08256/) (2020)
- [Adaptive Data Transport Mechanism for UAV Surveillance Missions in Lossy Environments](https://mlsanity.com/paper/2410.10843/) (2024)
- [All You Need for Object Detection: From Pixels, Points, and Prompts to Next-Gen Fusion and Multimodal LLMs/VLMs in Autonomous Vehicles](https://mlsanity.com/paper/2510.26641/) (2025)
- [Static-Equilibrium Oriented Interaction Force Modeling and Control of Aerial Manipulation with Uni-Directional Thrust Multirotors](https://mlsanity.com/paper/2306.12167/) (2023)
- [Safe Aerial Manipulator Maneuvering and Force Exertion via Control Barrier Functions](https://mlsanity.com/paper/2309.07709/) (2023)
- [ARDDQN: Attention Recurrent Double Deep Q-Network for UAV Coverage Path Planning and Data Harvesting](https://mlsanity.com/paper/2405.11013/) (2024)
- [Hybrid Fusion for Battery Degradation Diagnostics Using Minimal Real-World Data: Bridging Laboratory and Practical Applications](https://mlsanity.com/paper/2410.06469/) (2024)
- [Graph Based Deep Reinforcement Learning Aided by Transformers for Multi-Agent Cooperation](https://mlsanity.com/paper/2504.08195/) (2025)
- [From Shadow to Light: Toward Safe and Efficient Policy Learning Across MPC, DeePC, RL, and LLM Agents](https://mlsanity.com/paper/2510.04076/) (2025)

## Abstract (arXiv)

Drones equipped with overhead manipulators offer unique capabilities for inspection, maintenance, and contact-based interaction. However, the motion of the drone and its manipulator is tightly linked, and even small attitude changes caused by wind or control imperfections shift the end-effector away from its intended path. This coupling makes reliable tracking difficult and also limits the direct use of learning-based arm controllers that were originally designed for fixed-base robots. These effects appear consistently in our tests whenever the UAV body experiences drift or rapid attitude corrections. To address this behavior, we develop a reinforcement-learning (RL) framework with a transformer-based double deep Q learning (DDQN), with the core idea of using an adaptive beam-search planner that applies a short-horizon beam search over candidate control sequences using the learned critic as the forward estimator. This allows the controller to anticipate the end-effector's motion through simulated rollouts rather than executing those actions directly on the actual model, realizing a software-in-the-loop (SITL) approach. The lookahead relies on value estimates from a Transformer critic that processes short sequences of states, while a DDQN backbone provides the one-step targets needed to keep the learning process stable. Evaluated on a 3-DoF aerial manipulator under identical training conditions, the proposed meta-adaptive planner shows the strongest overall performance with a 10.2% reward increase, a substantial reduction in mean tracking error (from about 6% to 3%), and a 29.6% improvement in the combined reward-error metric relative to the DDQN baseline. Our method exhibits elevated stability in tracking target tip trajectory (by maintaining 5 cm tracking error) when the drone base exhibits drifts due to external disturbances, as opposed to the fixed-beam and Transformer-only variants.