# AgenticRL: Agentic Reinforcement Learning with Self-Refinement for Complex UAV Navigation

arXiv: https://arxiv.org/abs/2606.03963 · HTML: https://mlsanity.com/paper/2606.03963/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [Text2Reward: Reward Shaping with Language Models for Reinforcement Learning](https://mlsanity.com/paper/2309.11489/) (2023)
- [Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Control](https://mlsanity.com/paper/2103.02142/) (2021)
- [Active Preference-Based Gaussian Process Regression for Reward Learning](https://mlsanity.com/paper/2005.02575/) (2020)
- [AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework](https://mlsanity.com/paper/2510.04206/) (2025)
- [ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations](https://mlsanity.com/paper/2505.10911/) (2025)
- [RoboReward: General-Purpose Vision-Language Reward Models for Robotics](https://mlsanity.com/paper/2601.00675/) (2026)
- [Agentic Reinforcement Learning with Implicit Step Rewards](https://mlsanity.com/paper/2509.19199/) (2025)
- [Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards](https://mlsanity.com/paper/2506.11425/) (2025)
- [Code as Reward: Empowering Reinforcement Learning with VLMs](https://mlsanity.com/paper/2402.04764/) (2024)
- [Exploring Reasoning Reward Model for Agents](https://mlsanity.com/paper/2601.22154/) (2026)
- [LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent](https://mlsanity.com/paper/2604.17931/) (2026)
- [Sim-to-Real Deep Reinforcement Learning based Obstacle Avoidance for UAVs under Measurement Uncertainty](https://mlsanity.com/paper/2303.07243/) (2023)
- [OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning](https://mlsanity.com/paper/2510.24636/) (2025)
- [Reinforcement Learning for UAV control with Policy and Reward Shaping](https://mlsanity.com/paper/2212.03828/) (2022)

## Abstract (arXiv)

Deep reinforcement learning enables autonomous robots to learn complex navigation tasks, but still relies heavily on time consuming manual reward design and fine tuning. Existing automated reward generation and refinement methods reduce this effort, yet often lack task-level behavioral diagnosis for directing subsequent reward revisions. We introduce AgenticRL, a multimodal closed loop framework in which role-specialized agents generate executable rewards, diagnose failures of the resulting policies, formulate targeted refinement instructions, and regenerate improved rewards. Before training, a task grounding stage automatically selects a compatible action profile, together with its observation and reward interfaces. Each generated reward is used to train a policy using Proximal Policy Optimization (PPO), which is subsequently evaluated under randomized conditions. Task-level behavioral, geometric, and safety measurements are organized into a structured diagnosis packet and jointly analyzed with the current reward code, task specification, behavioral summary, and visual scene context. Unlike one-shot reward generation, human-guided refinement, or broad candidate search, AgenticRL uses automated diagnosis of the behavior induced by a reward to direct its next revision. We evaluate the framework across eight UAV tasks covering navigation, obstacle interaction, trajectory tracking, agile manoeuvres, and cluttered flight. Under the reported comparative evaluation, AgenticRL achieves success rates of 100% in racing and 88% in cluttered navigation, exceeding the strongest Eureka and Text2Reward baselines, respectively. Reward refinement increases mean simulation success from 37.2% to 96.4%, while the resulting policies achieve a collective real-world success rate of 90.0% and a sim-to-real accuracy of 93.4%.