# Mobile Robot Navigation Using Hand-Drawn Maps: A Vision Language Model Approach

arXiv: https://arxiv.org/abs/2502.00114 · HTML: https://mlsanity.com/paper/2502.00114/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 52 references that are in the map.
- First posted on arXiv 31 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 19 of 19)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions](https://mlsanity.com/paper/2309.11382/) (2023)
- [Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning](https://mlsanity.com/paper/2310.10103/) (2023)
- [Memory-Augmented Reinforcement Learning for Image-Goal Navigation](https://mlsanity.com/paper/2101.05181/) (2021)
- [A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models](https://mlsanity.com/paper/2308.07997/) (2023)
- [LangNav: Language as a Perceptual Representation for Navigation](https://mlsanity.com/paper/2310.07889/) (2023)
- [NavFormer: A Transformer Architecture for Robot Target-Driven Navigation in Unknown and Dynamic Environments](https://mlsanity.com/paper/2402.06838/) (2024)
- [Deep Reinforcement Learning for Decentralized Multi-Robot Exploration With Macro Actions](https://mlsanity.com/paper/2110.02181/) (2021)
- [Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systems](https://mlsanity.com/paper/2407.08275/) (2024)
- [High-Speed Robot Navigation using Predicted Occupancy Maps](https://mlsanity.com/paper/2012.12142/) (2020)
- [Robot Navigation with Map-Based Deep Reinforcement Learning](https://mlsanity.com/paper/2002.04349/) (2020)
- [TINA: Think, Interaction, and Action Framework for Zero-Shot Vision Language Navigation](https://mlsanity.com/paper/2403.08833/) (2024)
- [MLLM-Search: A Zero-Shot Approach to Finding People using Multimodal Large Language Models](https://mlsanity.com/paper/2412.00103/) (2024)
- [4CNet: A Diffusion Approach to Map Prediction for Decentralized Multi-Robot Exploration](https://mlsanity.com/paper/2402.17904/) (2024)
- [Monte Carlo Localization in Hand-Drawn Maps](https://mlsanity.com/paper/1504.00522/) (2015)

## Built on by (most cited first, 5 of 5)

- [Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Review](https://mlsanity.com/paper/2505.20503/) (2025)
- [SeqWalker: Sequential-Horizon Vision-and-Language Navigation with Hierarchical Planning](https://mlsanity.com/paper/2601.04699/) (2026)
- [Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models](https://mlsanity.com/paper/2505.07815/) (2025)
- [SketchPlan: Diffusion Based Drone Planning From Human Sketches](https://mlsanity.com/paper/2510.03545/) (2025)
- [SpatialFly: Implicit 3D Prior-Guided Visual Reparameterization for Continuous UAV Vision-and-Language Navigation](https://mlsanity.com/paper/2603.21046/) (2026)

## Abstract (arXiv)

Hand-drawn maps can be used to convey navigation instructions between humans and robots in a natural and efficient manner. However, these maps can often contain inaccuracies such as scale distortions and missing landmarks which present challenges for mobile robot navigation. This paper introduces a novel Hand-drawn Map Navigation (HAM-Nav) architecture that leverages pre-trained vision language models (VLMs) for robot navigation across diverse environments, hand-drawing styles, and robot embodiments, even in the presence of map inaccuracies. HAM-Nav integrates a unique Selective Visual Association Prompting approach for topological map-based position estimation and navigation planning as well as a Predictive Navigation Plan Parser to infer missing landmarks. Extensive experiments were conducted in photorealistic simulated environments, using both wheeled and legged robots, demonstrating the effectiveness of HAM-Nav in terms of navigation success rates and Success weighted by Path Length. Furthermore, a user study in real-world environments highlighted the practical utility of hand-drawn maps for robot navigation as well as successful navigation outcomes compared against a non-hand-drawn map approach.