# CoNVOI: Context-aware Navigation using Vision Language Models in Outdoor and Indoor Environments

arXiv: https://arxiv.org/abs/2403.15637 · HTML: https://mlsanity.com/paper/2403.15637/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 53 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Foundation Models in Robotics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2312.07843/) (2023)
- [PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs](https://mlsanity.com/paper/2402.07872/) (2024)
- [Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning](https://mlsanity.com/paper/1709.10082/) (2017)
- [A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models](https://mlsanity.com/paper/2308.07997/) (2023)
- [OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics](https://mlsanity.com/paper/2401.12202/) (2024)
- [TNS: Terrain Traversability Mapping and Navigation System for Autonomous Excavators](https://mlsanity.com/paper/2109.06250/) (2021)
- [Complex Terrain Navigation via Model Error Prediction](https://mlsanity.com/paper/2111.09768/) (2021)

## Built on by (most cited first, 30 of 43)

- [CityWalker: Learning Embodied Urban Navigation from Web-Scale Videos](https://mlsanity.com/paper/2411.17820/) (2024)
- [OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigation](https://mlsanity.com/paper/2509.19480/) (2025)
- [VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Model](https://mlsanity.com/paper/2410.08792/) (2024)
- [End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering](https://mlsanity.com/paper/2411.05755/) (2024)
- [Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigation](https://mlsanity.com/paper/2502.14254/) (2025)
- [CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models](https://mlsanity.com/paper/2508.13446/) (2025)
- [Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models](https://mlsanity.com/paper/2409.12969/) (2024)
- [OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization](https://mlsanity.com/paper/2506.07570/) (2025)
- [Object Detection with Multimodal Large Vision-Language Models: An In-depth Review](https://mlsanity.com/paper/2508.19294/) (2025)
- [VL-TGS: Trajectory Generation and Selection using Vision Language Models in Mapless Outdoor Environments](https://mlsanity.com/paper/2408.02454/) (2024)
- [SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigation](https://mlsanity.com/paper/2509.08757/) (2025)
- [Robot Navigation Using Physically Grounded Vision-Language Models in Outdoor Environments](https://mlsanity.com/paper/2409.20445/) (2024)
- [Multi-Agent LLM Actor-Critic Framework for Social Robot Navigation](https://mlsanity.com/paper/2503.09758/) (2025)
- [TANGO: Traversability-Aware Navigation with Local Metric Control for Topological Goals](https://mlsanity.com/paper/2509.08699/) (2025)
- [Vision Language Models Can Parse Floor Plan Maps](https://mlsanity.com/paper/2409.12842/) (2024)
- [Select2Plan: Training-Free ICL-Based Planning through VQA and Memory Retrieval](https://mlsanity.com/paper/2411.04006/) (2024)
- [ViLAM: Distilling Vision-Language Reasoning into Attention Maps for Social Robot Navigation](https://mlsanity.com/paper/2503.09820/) (2025)
- [From Obstacles to Etiquette: Robot Social Navigation with VLM-Informed Path Selection](https://mlsanity.com/paper/2602.09002/) (2026)
- [Improving Robotic Generalist Policies via Flow Reversal Steering](https://mlsanity.com/paper/2606.13675/) (2026)
- [Long-Term Planning Around Humans in Domestic Environments with 3D Scene Graphs](https://mlsanity.com/paper/2503.09173/) (2025)
- [ComposableNav: Instruction-Following Navigation in Dynamic Environments via Composable Diffusion](https://mlsanity.com/paper/2509.17941/) (2025)
- [MUSON: A Reasoning-oriented Multimodal Dataset for Socially Compliant Navigation in Urban Environments](https://mlsanity.com/paper/2512.22867/) (2025)
- [ZeST: A VLM-based Zero-Shot Traversability Navigation for Unknown Environments](https://mlsanity.com/paper/2508.19131/) (2025)
- [Beyond Recognition: Evaluating Visual Perspective Taking in Vision Language Models](https://mlsanity.com/paper/2505.03821/) (2025)
- [AnyTraverse: An off-road traversability framework with VLM and human operator in the loop](https://mlsanity.com/paper/2506.16826/) (2025)
- [VENTURA: Adapting Image Diffusion Models for Unified Task Conditioned Navigation](https://mlsanity.com/paper/2510.01388/) (2025)
- [SoraNav: Adaptive UAV Task-Centric Navigation via Zeroshot VLM Reasoning](https://mlsanity.com/paper/2510.25191/) (2025)
- [LISN: Language-Instructed Social Navigation with VLM-based Controller Modulating](https://mlsanity.com/paper/2512.09920/) (2025)
- [Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigation](https://mlsanity.com/paper/2606.01565/) (2026)
- [A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigation](https://mlsanity.com/paper/2607.09792/) (2026)

## Abstract (arXiv)

We present ConVOI, a novel method for autonomous robot navigation in real-world indoor and outdoor environments using Vision Language Models (VLMs). We employ VLMs in two ways: first, we leverage their zero-shot image classification capability to identify the context or scenario (e.g., indoor corridor, outdoor terrain, crosswalk, etc) of the robot's surroundings, and formulate context-based navigation behaviors as simple text prompts (e.g. ``stay on the pavement"). Second, we utilize their state-of-the-art semantic understanding and logical reasoning capabilities to compute a suitable trajectory given the identified context. To this end, we propose a novel multi-modal visual marking approach to annotate the obstacle-free regions in the RGB image used as input to the VLM with numbers, by correlating it with a local occupancy map of the environment. The marked numbers ground image locations in the real-world, direct the VLM's attention solely to navigable locations, and elucidate the spatial relationships between them and terrains depicted in the image to the VLM. Next, we query the VLM to select numbers on the marked image that satisfy the context-based behavior text prompt, and construct a reference path using the selected numbers. Finally, we propose a method to extrapolate the reference trajectory when the robot's environmental context has not changed to prevent unnecessary VLM queries. We use the reference trajectory to guide a motion planner, and demonstrate that it leads to human-like behaviors (e.g. not cutting through a group of people, using crosswalks, etc.) in various real-world indoor and outdoor scenarios.