mlsanityData to 7 Oct 2026

Guide-LLM: An Embodied LLM Agent and Text-Based Topological Map for Robotic Guidance of People with Visual Impairments

arXiv 2410.20666 ↗First posted 28 Oct 2024cs.RO cs.AI cs.CLcited by 10

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Navigation presents a significant challenge for persons with visual impairments (PVI). While traditional aids such as white canes and guide dogs are invaluable, they fall short in delivering detailed spatial information and precise guidance to desired locations. Recent developments in large language models (LLMs) and vision-language models (VLMs) offer new avenues for enhancing assistive navigation. In this paper, we introduce Guide-LLM, an embodied LLM-based agent designed to assist PVI in navigating large indoor environments. Our approach features a novel text-based topological map that enables the LLM to plan global paths using a simplified environmental representation, focusing on straight paths and right-angle turns to facilitate navigation. Additionally, we utilize the LLM's commonsense reasoning for hazard detection and personalized path planning based on user preferences. Simulated experiments demonstrate the system's efficacy in guiding PVI, underscoring its potential as a significant advancement in assistive technology. The results highlight Guide-LLM's ability to offer efficient, adaptive, and personalized navigation assistance, pointing to promising advancements in this field.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 33 of its 59 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

33 of 59 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View12 Jul
  2. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation6 Mar
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents18 Jan

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  8. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  9. 2022Visual Language Maps for Robot Navigationcited by 362
  10. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  11. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  12. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
18 more
  1. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  2. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  3. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  4. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  5. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  6. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  7. 2023Self-planning Code Generation with Large Language Modelscited by 121
  8. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 116
  9. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  10. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
  11. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
  12. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  13. 2023SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environmentscited by 53
  14. 2020Soft Expert Reward Learning for Vision-and-Language Navigationcited by 37
  15. 2024RoboHop: Segment-based Topological Map Representation for Open-World Visual Navigationcited by 25
  16. 2023DRAGON: A Dialogue-Based Robot for Assistive Navigation with Visual Language Groundingcited by 9
  17. 2019Vision-to-Language Tasks Based on Attributes and Attention Mechanismcited by 3
  18. 2024Vision-based Wearable Steering Assistance for People with Impaired Vision in Joggingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 11
  2. 2025Assessing the Value of Visual Input: A Benchmark of Multimodal Large Language Models for Robotic Path Planningcited by 4
  3. 2025osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoningcited by 2
  4. 2026Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Userscited by 2
  5. 2026VIABLE: A Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluationcited by 1
  6. 2026Towards general embodied intelligence: integrating large language models, knowledge bases, and reasoning capabilities to build the next generation of AI agentscited by 1
  7. 2026HaltNav: Reactive Visual Halting over Lightweight Topological Priors for Robust Vision-Language Navigationcited by 0
  8. 2026Time-Aware Assistive Navigationcited by 0
  9. 2026Commonsense-Grounded Path Planning from Abstract Instructionscited by 0
  10. 2026Spatial Strategies, Not Actions: Vector-Quantized Geodesics as Tools for LLM-Driven Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LLM-Guided Agentic Floor Plan Parsing for Accessible Indoor Navigation of Blind and Low-Vision People
  2. 2025OpenGuide: Assistive Object Retrieval in Indoor Spaces for Individuals with Visual Impairments ↗
  3. 2024LLM-Enhanced Path Planning: Safe and Efficient Autonomous Navigation with Instructional Inputs
  4. 2026Assisted Spatial Cognition Through Vision-Language Models
  5. 2025Space-Aware Instruction Tuning: Dataset and Benchmark for Guide Dog Robots Assisting the Visually Impaired
  6. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models
  7. 2025Navigating Motion Agents in Dynamic and Cluttered Environments through LLM Reasoning
  8. 2025MARS: Multi-Agent Robotic System with Multimodal Large Language Models for Assistive Intelligence