mlsanityData to 7 Oct 2026

VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View

arXiv 2307.06082 ↗First posted 12 Jul 2023cs.AI cs.CL cs.CVcited by 67

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Incremental decision making in real-world environments is one of the most challenging tasks in embodied artificial intelligence. One particularly demanding scenario is Vision and Language Navigation~(VLN) which requires visual and natural language understanding as well as spatial and temporal reasoning capabilities. The embodied agent needs to ground its understanding of navigation instructions in observations of a real-world environment like Street View. Despite the impressive results of LLMs in other research areas, it is an ongoing problem of how to best connect them with an interactive visual environment. In this work, we propose VELMA, an embodied LLM agent that uses a verbalization of the trajectory and of visual environment observations as contextual prompt for the next action. Visual information is verbalized by a pipeline that extracts landmarks from the human written navigation instructions and uses CLIP to determine their visibility in the current panorama view. We show that VELMA is able to successfully follow navigation instructions in Street View with only two in-context examples. We further finetune the LLM agent on a few thousand examples and achieve 25%-30% relative improvement in task completion over the previous state-of-the-art for two datasets.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 42 references that are in the map. First posted on arXiv 12 Jul 2023.

Builds on

30 of 42 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  10. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  11. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  12. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
18 more
  1. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  2. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  3. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  4. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  5. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  6. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  7. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  8. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  9. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  10. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  11. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  12. 2020BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Stepscited by 49
  13. 2019Learning To Follow Directions in Street Viewcited by 29
  14. 2022Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areascited by 24
  15. 2020Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigationcited by 22
  16. 2020Generating Landmark Navigation Instructions from Maps as a Graph-to-Text Problemcited by 22
  17. 2020Learning to Stop: A Simple yet Effective Approach to Urban Vision-Language Navigationcited by 20
  18. 2022A Priority Map for Vision-and-Language Navigation with Trajectory Plans and Feature-Location Cuescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 26; 2025: 27; 2026: 10.

67 papers.

Sort:

By area:

  1. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  2. 2024Evaluating the World Model Implicit in a Generative Modelcited by 107
  3. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
  4. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  5. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  6. 2024V-IRL: Grounding Virtual Intelligence in Real Lifecited by 35
  7. 2024Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agentscited by 35
  8. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  9. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  10. 2025A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrievalcited by 31
  11. 2025Safety Guardrails for LLM-Enabled Robotscited by 30
  12. 2025CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memorycited by 30
38 more
  1. 2025SmartWay: Enhanced Waypoint Prediction and Backtracking for Zero-Shot Vision-and-Language Navigationcited by 29
  2. 2024UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarioscited by 28
  3. 2024CityBench: Evaluating the Capabilities of Large Language Models for Urban Taskscited by 26
  4. 2024MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chainscited by 25
  5. 2024Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructionscited by 25
  6. 2024Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Modelscited by 23
  7. 2025Seek-CAD: A Self-refined Generative Modeling for 3D Parametric CAD Using Local Inference via DeepSeekcited by 22
  8. 2024NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigationcited by 18
  9. 2024Policy Improvement using Language Feedback Modelscited by 16
  10. 2025MonoDream: Monocular Vision-Language Navigation with Panoramic Dreamingcited by 15
  11. 2023Advances in Embodied Navigation Using Large Language Models: A Surveycited by 14
  12. 2023Reasoning about the Unseen for Efficient Outdoor Object Navigationcited by 13
  13. 2023Towards Responsible Generative AI: A Reference Architecture for Designing Foundation Model based Agentscited by 13
  14. 2024A Survey on Human-Centric LLMscited by 13
  15. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 11
  16. 2024Never-Ending Behavior-Cloning Agent for Robotic Manipulationcited by 10
  17. 2024Guide-LLM: An Embodied LLM Agent and Text-Based Topological Map for Robotic Guidance of People with Visual Impairmentscited by 10
  18. 2025Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Reviewcited by 10
  19. 2025SE-VLN: A Self-Evolving Vision-Language Navigation Framework Based on Multimodal Large Language Modelscited by 9
  20. 2025V-Stylist: Video Stylization via Collaboration and Reflection of MLLM Agentscited by 8
  21. 2025CitySeeker: How Do VLMS Explore Embodied Urban Navigation With Implicit Human Needs?cited by 8
  22. 2024How Secure Are Large Language Models (LLMs) for Navigation in Urban Environments?cited by 7
  23. 2026textscNaVIDA: Vision-Language Navigation with Inverse Dynamics Augmentationcited by 5
  24. 2024MEIA: Multimodal Embodied Perception and Interaction in Unknown Environmentscited by 4
  25. 2025gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttlingcited by 4
  26. 2025EvolveNav: Empowering LLM-Based Vision-Language Navigation via Self-Improving Embodied Reasoningcited by 4
  27. 2025AgriVLN: Vision-and-Language Navigation for Agricultural Robotscited by 4
  28. 2025Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challengescited by 4
  29. 2024Cog-GA: A Large Language Models-based Generative Agent for Vision-Language Navigation in Continuous Environmentscited by 3
  30. 2024A Survey on Large Language Model-empowered Autonomous Drivingcited by 3
  31. 2025EmoAgent: A Multi-Agent Framework for Diverse Affective Image Manipulationcited by 3
  32. 2025Mem4Nav: Boosting Vision-and-Language Navigation in Urban Environments with a Hierarchical Spatial-Cognition Long-Short Memory Systemcited by 3
  33. 2025Compositional Coordination for Multi-Robot Teams with Large Language Modelscited by 3
  34. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  35. 2025UNeMo: Collaborative Visual-Language Reasoning and Navigation via a Multimodal World Modelcited by 3
  36. 2026Think Twice, Act Once: Verifier-Guided Action Selection For Embodied Agentscited by 3
  37. 2024Embodied AI in Mobile Robots: Coverage Path Planning with Large Language Modelscited by 2
  38. 2024Can LLMs plan paths in the real world?cited by 2

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024FLAME: Learning to Navigate with Multimodal LLM in Urban Environments
  2. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models
  3. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents
  4. 2025Vision-and-Language Navigation with Analogical Textual Descriptions in LLMs
  5. 2026LightNav-0: Eliciting VLM Spatial Intelligence for Generalist Embodied Navigation
  6. 2024End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering
  7. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation
  8. 2026AgentVLN: Towards Agentic Vision-and-Language Navigation