mlsanityData to 14 Oct 2026

WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigation

arXiv 2603.10703 ↗First posted 11 Mar 2026cs.CV cs.CYcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Ensuring accessible pedestrian navigation requires reasoning about both semantic and spatial aspects of complex urban scenes, a challenge that existing Large Vision-Language Models (LVLMs) struggle to meet. Although these models can describe visual content, their lack of explicit grounding leads to object hallucinations and unreliable depth reasoning, limiting their usefulness for accessibility guidance. We introduce WalkGPT, a pixel-grounded LVLM for the new task of Grounded Navigation Guide, unifying language reasoning and segmentation within a single architecture for depth-aware accessibility guidance. Given a pedestrian-view image and a navigation query, WalkGPT generates a conversational response with segmentation masks that delineate accessible and harmful features, along with relative depth estimation. The model incorporates a Multi-Scale Query Projector (MSQP) that shapes the final image tokens by aggregating them along text tokens across spatial hierarchies, and a Calibrated Text Projector (CTP), guided by a proposed Region Alignment Loss, that maps language embeddings into segmentation-aware representations. These components enable fine-grained grounding and depth inference without user-provided cues or anchor points, allowing the model to generate complete and realistic navigation guidance. We also introduce PAVE, a large-scale benchmark of 41k pedestrian-view images paired with accessibility-aware questions and depth-grounded answers. Experiments show that WalkGPT achieves strong grounded reasoning and segmentation performance. The source code and dataset are available on the hrefhttps://sites.google.com/view/walkgpt-26/homeproject website.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 62 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

54 of 62 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos7 Jan
  2. 2024WalkVLM:Aid Visually Impaired People Walking by Vision Language Model30 Dec
  3. 2024F-LMM: Grounding Frozen Large Multimodal Models9 Jun
  4. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models19 Apr
  5. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model21 Mar
  6. 2023GSVA: Generalized Segmentation via Multimodal Large Language Models15 Dec
  7. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models5 Dec
  8. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  9. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  10. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  11. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 42

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2023Segment Anythingcited by 8,234
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  11. 2022Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Imagescited by 830
  12. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
30 more
  1. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  2. 2023Segment Everything Everywhere All at Oncecited by 548
  3. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  4. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  5. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensingcited by 328
  6. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  7. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  8. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 260
  9. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  10. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  11. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  12. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  13. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areascited by 95
  14. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 81
  15. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  16. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 63
  17. 2025GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensingcited by 57
  18. 2024LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoningcited by 52
  19. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMscited by 47
  20. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 46
  21. 2025DepthLM: Metric Depth From Vision Language Modelscited by 30
  22. 2025MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal outputcited by 15
  23. 2024Is it safe to cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossingcited by 12
  24. 2026SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?cited by 11
  25. 2023SANPO: A Scene Understanding, Accessibility and Human Navigation Datasetcited by 10
  26. 2023GeoSAM: Fine-tuning SAM with Multi-Modal Prompts for Mobility Infrastructure Segmentationcited by 7
  27. 2024StreetviewLLM: Extracting Geographic Information Using a Chain-of-Thought Multimodal Large Language Modelcited by 6
  28. 2025Enhancing Spatial Reasoning in Multimodal Large Language Models through Reasoning-based Segmentationcited by 6
  29. 2025StreetReaderAI: Making Street View Accessible Using Context-Aware Multimodal AIcited by 5
  30. 2025mmWalk: Towards Multi-modal Multi-view Walking Assistancecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Modelscited by 0
  2. 2026Soft Spatial Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Green for Go, Red for No: Visual Grounding via Semantic Segmentation for VLA Navigation Policies
  2. 2024CoNVOI: Context-aware Navigation using Vision Language Models in Outdoor and Indoor Environments
  3. 2026Traj-VLN: Learning Pixel-Space Interaction via Autoregressive Trajectory Generation
  4. 2025GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance
  5. 2026Grounded 3D-Aware Spatial Vision-Language Modeling
  6. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
  7. 2025OmniPT: Unleashing the Potential of Large Vision Language Models for Pedestrian Tracking and Understanding
  8. 2026PEDESTRIANQA: A Benchmark for Vision-Language Models on Pedestrian Intention and Trajectory Prediction