# History-Augmented Vision-Language Models for Frontier-Based Zero-Shot Object Navigation

arXiv: https://arxiv.org/abs/2506.16623 · HTML: https://mlsanity.com/paper/2506.16623/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 24 references that are in the map.
- First posted on arXiv 19 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [NVILA: Efficient Frontier Visual Language Models](https://mlsanity.com/paper/2412.04468/) (2024)
- [VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation](https://mlsanity.com/paper/2312.03275/) (2023)
- [ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings](https://mlsanity.com/paper/2206.12403/) (2022)
- [L3MVN: Leveraging Large Language Models for Visual Target Navigation](https://mlsanity.com/paper/2304.05501/) (2023)
- [Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models](https://mlsanity.com/paper/2407.07035/) (2024)
- [Bridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skill](https://mlsanity.com/paper/2309.10309/) (2023)
- [CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation](https://mlsanity.com/paper/2211.16649/) (2022)
- [ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination](https://mlsanity.com/paper/2410.09874/) (2024)
- [GAMap: Zero-Shot Object Goal Navigation with Multi-Scale Geometric-Affordance Guidance](https://mlsanity.com/paper/2410.23978/) (2024)
- [VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement](https://mlsanity.com/paper/2210.05064/) (2022)
- [FrontierNet: Learning Visual Cues to Explore](https://mlsanity.com/paper/2501.04597/) (2025)
- [Object Goal Navigation using Data Regularized Q-Learning](https://mlsanity.com/paper/2208.13009/) (2022)
- [Frontier Based Exploration for Autonomous Robot](https://mlsanity.com/paper/1806.03581/) (2018)

## Built on by (most cited first, 4 of 4)

- [AREA3D: Active Reconstruction Agent with Unified Feed-Forward 3D Perception and Vision-Language Guidance](https://mlsanity.com/paper/2512.05131/) (2025)
- [edgeVLM: Cloud-edge Collaborative Real-time VLM based on Context Transfer](https://mlsanity.com/paper/2508.12638/) (2025)
- [OpenFrontier: General Navigation with Visual-Language Grounded Frontiers](https://mlsanity.com/paper/2603.05377/) (2026)
- [LIME: Learning Intent-aware Camera Motion from Egocentric Video](https://mlsanity.com/paper/2607.02417/) (2026)

## Abstract (arXiv)

Object Goal Navigation (ObjectNav) challenges robots to find objects in unseen environments, demanding sophisticated reasoning. While Vision-Language Models (VLMs) show potential, current ObjectNav methods often employ them superficially, primarily using vision-language embeddings for object-scene similarity checks rather than leveraging deeper reasoning. This limits contextual understanding and leads to practical issues like repetitive navigation behaviors. This paper introduces a novel zero-shot ObjectNav framework that pioneers the use of dynamic, history-aware prompting to more deeply integrate VLM reasoning into frontier-based exploration. Our core innovation lies in providing the VLM with action history context, enabling it to generate semantic guidance scores for navigation actions while actively avoiding decision loops. We also introduce a VLM-assisted waypoint generation mechanism for refining the final approach to detected objects. Evaluated on the HM3D dataset within Habitat, our approach achieves a 46% Success Rate (SR) and 24.8% Success weighted by Path Length (SPL). These results are comparable to state-of-the-art zero-shot methods, demonstrating the significant potential of our history-augmented VLM prompting strategy for more robust and context-aware robotic navigation.