mlsanityData to 14 Oct 2026

PGN: Design and Implementation of a Vision-Language Navigation System Based on Pangu Multimodal Foundation Model

arXiv 2607.17806 ↗First posted 20 Jul 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Vision-Language Navigation (VLN) requires an embodied agent to interpret a natural-language instruction and predict actions from temporally ordered visual observations. Adapting a multimodal large language model to VLN requires visual-language alignment, compact temporal inputs, action-space grounding, and stable training on the target hardware. This technical report presents PGN (Pangu Navigator), an offline VLN action-prediction system built on OpenPangu-7B. Training proceeds in two stages. First, PGMM aligns a frozen EVA-ViT-G/14 vision encoder with the frozen language backbone by training a Q-Former and a two-layer MLP projector. Second, PGN adapts the aligned model to expert navigation trajectories using five-observation windows, epoch-dependent temporal sampling, and a reasoning-then-action output format; this stage freezes the aligned visual pathway and updates three structural-token embeddings and LoRA adapters. The implementation combines mixed-precision computation, selective FP32 computation, and DeepSpeed ZeRO-2 on eight Ascend 910B NPUs. Under teacher-forced, open-loop evaluation on 500 held-out expert trajectories, V9 reports a 62.29% Normalized Action Match (NAM) and a 100.00% Non-empty Rate (NER). These metrics quantify offline expert-action alignment rather than closed-loop navigation success; evaluating error accumulation, path efficiency, and goal completion remains future work.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 16 references that are in the map. First posted on arXiv 20 Jul 2026.

Builds on

12 of 16 references.

References · 12

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  6. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  7. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  8. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  9. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  10. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  11. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  12. 2025Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognitioncited by 62

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Navigation Framework Utilizing Vision-Language Models
  2. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning
  3. 2025Efficient-VLN: A Simple yet Strong Baseline for Efficient Vision-Language Navigation
  4. 2026GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model
  5. 2026LookStep: Efficient Vision-Language Navigation with Linguistic Foresight and Event Driven Memory
  6. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Tasks
  7. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation
  8. 2026SFCo-Nav: Efficient Zero-Shot Visual Language Navigation via Collaboration of Slow LLM and Fast Attributed Graph Alignment