# HEADER: Hierarchical Robot Exploration via Attention-Based Deep Reinforcement Learning with Expert-Guided Reward

arXiv: https://arxiv.org/abs/2510.15679 · HTML: https://mlsanity.com/paper/2510.15679/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 30 of its 58 references that are in the map.
- First posted on arXiv 17 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [From Louvain to Leiden: guaranteeing well-connected communities](https://mlsanity.com/paper/1810.08473/) (2018)
- [LIO-SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping](https://mlsanity.com/paper/2007.00258/) (2020)
- [FAST-LIO2: Fast Direct LiDAR-inertial Odometry](https://mlsanity.com/paper/2107.06829/) (2021)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Soft Actor-Critic for Discrete Action Settings](https://mlsanity.com/paper/1910.07207/) (2019)
- [Learning High-Speed Flight in the Wild](https://mlsanity.com/paper/2110.05113/) (2021)
- [Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards](https://mlsanity.com/paper/1911.01417/) (2019)
- [Self-Supervised Online Reward Shaping in Sparse-Reward Environments](https://mlsanity.com/paper/2103.04529/) (2021)
- [Uncertainty-driven Planner for Exploration and Navigation](https://mlsanity.com/paper/2202.11907/) (2022)
- [Autonomous Exploration Under Uncertainty via Deep Reinforcement Learning on Graphs](https://mlsanity.com/paper/2007.12640/) (2020)
- [Autonomous Exploration Development Environment and the Planning Algorithms](https://mlsanity.com/paper/2110.14573/) (2021)
- [ARiADNE: A Reinforcement learning approach using Attention-based Deep Networks for Exploration](https://mlsanity.com/paper/2301.11575/) (2023)
- [Deep Reinforcement Learning-based Large-scale Robot Exploration](https://mlsanity.com/paper/2403.10833/) (2024)
- [Autonomous UAV Exploration of Dynamic Environments via Incremental Sampling and Probabilistic Roadmap](https://mlsanity.com/paper/2010.07429/) (2020)
- [MapEx: Indoor Structure Exploration with Probabilistic Information Gain from Global Map Predictions](https://mlsanity.com/paper/2409.15590/) (2024)
- [Learning Vision-based Pursuit-Evasion Robot Policies](https://mlsanity.com/paper/2308.16185/) (2023)
- [FSMI: Fast computation of Shannon Mutual Information for information-theoretic mapping](https://mlsanity.com/paper/1905.02238/) (2019)
- [Explore-Bench: Data Sets, Metrics and Evaluations for Frontier-based and Deep-reinforcement-learning-based Autonomous Exploration](https://mlsanity.com/paper/2202.11931/) (2022)
- [FIG-OP: Exploring Large-Scale Unknown Environments on a Fixed Time Budget](https://mlsanity.com/paper/2203.06316/) (2022)
- [MAexp: A Generic Platform for RL-based Multi-Agent Exploration](https://mlsanity.com/paper/2404.12824/) (2024)
- [HDPlanner: Advancing Autonomous Deployments in Unknown Environments through Hierarchical Decision Networks](https://mlsanity.com/paper/2408.03768/) (2024)
- [IR2: Implicit Rendezvous for Robotic Exploration Teams under Sparse Intermittent Connectivity](https://mlsanity.com/paper/2409.04730/) (2024)
- [Active Mapping via Gradient Ascent Optimization of Shannon Mutual Information over Continuous SE(3) Trajectories](https://mlsanity.com/paper/2204.07623/) (2022)
- [HPHS: Hierarchical Planning based on Hybrid Frontier Sampling for Unknown Environments Exploration](https://mlsanity.com/paper/2407.10660/) (2024)

## Built on by (most cited first, 1 of 1)

- [FARE: Fast-Slow Agentic Robotic Exploration](https://mlsanity.com/paper/2601.14681/) (2026)

## Abstract (arXiv)

This work pushes the boundaries of learning-based methods in autonomous robot exploration in terms of environmental scale and exploration efficiency. We present HEADER, an attention-based reinforcement learning approach with hierarchical graphs for efficient exploration in large-scale environments. HEADER follows existing conventional methods to construct hierarchical representations for the robot belief/map, but further designs a novel community-based algorithm to construct and update a global graph, which remains fully incremental, shape-adaptive, and operates with linear complexity. Building upon attention-based networks, our planner finely reasons about the nearby belief within the local range while coarsely leveraging distant information at the global scale, enabling next-best-viewpoint decisions that consider multi-scale spatial dependencies. Beyond novel map representation, we introduce a parameter-free privileged reward that significantly improves model performance and produces near-optimal exploration behaviors, by avoiding training objective bias caused by handcrafted reward shaping. In simulated challenging, large-scale exploration scenarios, HEADER demonstrates better scalability than most existing learning and non-learning methods, while achieving a significant improvement in exploration efficiency (up to 20%) over state-of-the-art baselines. We also deploy HEADER on hardware and validate it in complex, large-scale real-life scenarios, including a 300m*230m campus environment.