# GeoScaffold: Learning Compact Geometric Latents via Reconstruction for Efficient Vision-Language Navigation

arXiv: https://arxiv.org/abs/2610.02697 · HTML: https://mlsanity.com/paper/2610.02697/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 53 of its 54 references that are in the map.
- First posted on arXiv 2 Oct 2026.

## Builds on (most cited first, 30 of 53)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [BLINK: Multimodal Large Language Models Can See but Not Perceive](https://mlsanity.com/paper/2404.12390/) (2024)
- [Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2004.02857/) (2020)
- [Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding](https://mlsanity.com/paper/2010.07954/) (2020)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation](https://mlsanity.com/paper/2402.15852/) (2024)
- [DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge](https://mlsanity.com/paper/2507.04447/) (2025)
- [NaVILA: Legged Robot Vision-Language-Action Model for Navigation](https://mlsanity.com/paper/2412.04453/) (2024)
- [Learning to Navigate in Complex Environments](https://mlsanity.com/paper/1611.03673/) (2016)
- [Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks](https://mlsanity.com/paper/1911.07883/) (2019)
- [InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environment](https://mlsanity.com/paper/2406.04882/) (2024)
- [Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks](https://mlsanity.com/paper/2412.06224/) (2024)
- [Towards Learning a Generalist Model for Embodied Navigation](https://mlsanity.com/paper/2312.02010/) (2023)
- [Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Model](https://mlsanity.com/paper/2510.12276/) (2025)
- [Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.02764/) (2022)
- [GridMM: Grid Memory Map for Vision-and-Language Navigation](https://mlsanity.com/paper/2307.12907/) (2023)
- [StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling](https://mlsanity.com/paper/2507.05240/) (2025)
- [Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2109.15207/) (2021)

## Abstract (arXiv)

Recent vision-and-language navigation (VLN) systems increasingly adopt streaming Video-LLM policies that map egocentric RGB observations and instructions directly to low-level actions. Yet these policies inherit weak 3D geometric priors from 2D pretraining. Existing geometry-aware extensions charge a persistent inference-time price: depth sensors, 3D encoders, or per-step perception tool calls. We propose GeoScaffold, a geometric supervision framework that pays this price once, at training time, by internalizing geometry into the policy itself. It first learns a compact depth tokenizer on depth maps from the training trajectories and freezes it. It then fine-tunes the policy with a handful of learnable geometry query tokens, training their hidden states to reconstruct navigation-critical geometry such as depth, connectivity, and traversability. This supervision turns the query states into compact geometric latents for action decoding, and through the shared weights also internalizes geometry into the backbone's own representations. Like a scaffold, the tokenizer, target generators, and reconstruction heads are discarded after training, leaving the backbone and action interface unchanged. Extensive experiments show that GeoScaffold consistently outperforms leading vision-only navigators on continuous VLN benchmarks, offering a practical paradigm for lightweight edge deployment of spatially aware embodied navigation models.