mlsanityData to 14 Oct 2026

Instruction-as-State: Environment-Guided and State-Conditioned Semantic Understanding for Embodied Navigation

arXiv 2604.18223 ↗First posted 20 Apr 2026cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Vision-and-Language Navigation requires agents to follow natural-language instructions in visually changing environments. A central challenge is the dynamic entanglement between language and observations: the meaning of instruction shifts as the agent's field of view and spatial context evolve. However, many existing models encode the instruction as a static global representation, limiting their ability to adapt instruction meaning to the current visual context. We therefore model instruction understanding as an Instruction-as-State variable: a decision-relevant, token-level instruction state that evolves step by step conditioned on the agent's perceptual state, where the perceptual state denotes the observation-grounded navigation context at each step. To realize this principle, we introduce State-Entangled Environment-Guided Instruction Understanding (S-EGIU), a coarse-to-fine framework for state-conditioned segment activation and token-level semantic refinement. At the coarse level, S-EGIU activates the instruction segment whose semantics align with the current observation. At the fine level, it refines the activated segment through observation-guided token grounding and contextual modeling, sharpening its internal semantics under the current observation. Together, these stages maintain an instruction state that is continuously updated according to the agent's perceptual state during navigation. S-EGIU delivers strong performance on several key metrics, including a +2.68% SPL gain on REVERIE Test Unseen, and demonstrates consistent efficiency gains across multiple VLN benchmarks, underscoring the value of dynamic instruction--perception entanglement.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

36 of 50 references.

References · 36

  1. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  2. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  3. 2022Visual Language Maps for Robot Navigationcited by 365
  4. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  5. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  6. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  7. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  8. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  9. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  10. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
  11. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 176
  12. 2021Episodic Transformer for Vision-and-Language Navigationcited by 156
24 more
  1. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 119
  2. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  3. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  4. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  5. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  6. 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigationcited by 71
  7. 2020Sub-Instruction Aware Vision-and-Language Navigationcited by 66
  8. 2025MapNav: A Novel Memory Representation via Annotated Semantic Maps for Vision-and-Language Navigationcited by 60
  9. 2021SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigationcited by 53
  10. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  11. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  12. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  13. 2021Vision-Language Navigation: A Survey and Taxonomycited by 34
  14. 2023VLN-PETL: Parameter-Efficient Transfer Learning for Vision-and-Language Navigationcited by 23
  15. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 19
  16. 2021Understanding Mobile GUI: from Pixel-Words to Screen-Sentencescited by 13
  17. 2025REGNav: Room Expert Guided Image-Goal Navigationcited by 12
  18. 2023Breaking through the learning plateaus of in-context learning in Transformercited by 8
  19. 2026PatchCue: Enhancing Vision-Language Model Reasoning with Patch-Based Visual Cuescited by 8
  20. 2019Recognition Of Surface Defects On Steel Sheet Using Transfer Learningcited by 7
  21. 2024Semantic-aware Representation Learning for Homography Estimationcited by 7
  22. 2024Camera-aware Label Refinement for Unsupervised Person Re-identificationcited by 4
  23. 2025TRAVEL: Training-Free Retrieval and Alignment for Vision-and-Language Navigationcited by 3
  24. 2024World-Consistent Data Generation for Vision-and-Language Navigationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026ELVA: Exploring Ranking-Driven Universal Multimodal Retrievalcited by 5
  2. 2026Overcoming the Weakest-Link Effect in LLM-Driven Program Optimization via Heterogeneous Edit Recombinationcited by 2
  3. 2026GraphIR: Architecture-Level Search States for LLM-Guided Neural Architecture Evolutioncited by 2
  4. 2026An AI4AI Framework for Visual Token Pruningcited by 1