# Deep Reinforcement Learning Based Navigation with Macro Actions and Topological Maps

arXiv: https://arxiv.org/abs/2504.18300 · HTML: https://mlsanity.com/paper/2504.18300/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 11 of 11)

- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Semi-parametric Topological Memory for Navigation](https://mlsanity.com/paper/1803.00653/) (2018)
- [Learning to Navigate in Complex Environments](https://mlsanity.com/paper/1611.03673/) (2016)
- [MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation](https://mlsanity.com/paper/2012.03912/) (2020)
- [Neural Map: Structured Memory for Deep Reinforcement Learning](https://mlsanity.com/paper/1702.08360/) (2017)
- [Successor Options: An Option Discovery Framework for Reinforcement Learning](https://mlsanity.com/paper/1905.05731/) (2019)
- [TopoNav: Topological Navigation for Efficient Exploration in Sparse Reward Environments](https://mlsanity.com/paper/2402.04061/) (2024)
- [Leveraging Topological Maps in Deep Reinforcement Learning for Multi-Object Navigation](https://mlsanity.com/paper/2310.10250/) (2023)

## Built on by (most cited first, 4 of 4)

- [Graph-Enhanced Policy Optimization in LLM Agent Training](https://mlsanity.com/paper/2510.26270/) (2025)
- [Meta-learning how to Share Credit among Macro-Actions](https://mlsanity.com/paper/2506.13690/) (2025)
- [AgenticDiffusion: Multi-View Reasoning with View-Conditioned Diffusion Planning for Vision-Based UAV Navigation](https://mlsanity.com/paper/2606.04111/) (2026)
- [Macro-Action Topological Navigation under Noisy Localization using Reinforcement Learning](https://mlsanity.com/paper/2608.23055/) (2026)

## Abstract (arXiv)

This paper addresses the challenge of navigation in large, visually complex environments with sparse rewards. We propose a method that uses object-oriented macro actions grounded in a topological map, allowing a simple Deep Q-Network (DQN) to learn effective navigation policies. The agent builds a map by detecting objects from RGBD input and selecting discrete macro actions that correspond to navigating to these objects. This abstraction drastically reduces the complexity of the underlying reinforcement learning problem and enables generalization to unseen environments. We evaluate our approach in a photorealistic 3D simulation and show that it significantly outperforms a random baseline under both immediate and terminal reward conditions. Our results demonstrate that topological structure and macro-level abstraction can enable sample-efficient learning even from pixel data.