# GroundControl: Anticipating Navigation Failures in Vision-Language Agents via Trajectory-Consistent Uncertainty Estimates

arXiv: https://arxiv.org/abs/2606.20479 · HTML: https://mlsanity.com/paper/2606.20479/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 24 references that are in the map.
- First posted on arXiv 18 Jun 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 20 of 20)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2004.02857/) (2020)
- [Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding](https://mlsanity.com/paper/2010.07954/) (2020)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents](https://mlsanity.com/paper/2502.09560/) (2025)
- [Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks](https://mlsanity.com/paper/1911.07883/) (2019)
- [Auxiliary Tasks and Exploration Enable ObjectNav](https://mlsanity.com/paper/2104.04112/) (2021)
- [Embodied Navigation Foundation Model](https://mlsanity.com/paper/2509.12129/) (2025)
- [Learning Conformal Abstention Policies for Adaptive Risk Management in Large Language and Vision-Language Models](https://mlsanity.com/paper/2502.06884/) (2025)
- [VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation](https://mlsanity.com/paper/2512.19021/) (2025)
- [NaviTrace: Evaluating Embodied Navigation of Vision-Language Models](https://mlsanity.com/paper/2510.26909/) (2025)
- [VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic Memory](https://mlsanity.com/paper/2601.08665/) (2026)
- [Vision-Language Navigation with Embodied Intelligence: A Survey](https://mlsanity.com/paper/2402.14304/) (2024)
- [Uncertainty in Action: Confidence Elicitation in Embodied Agents](https://mlsanity.com/paper/2503.10628/) (2025)
- [Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigation](https://mlsanity.com/paper/2411.07848/) (2024)
- [Embodied Navigation with Auxiliary Task of Action Description Prediction](https://mlsanity.com/paper/2510.21809/) (2025)
- [A Self-Supervised Auxiliary Loss for Deep RL in Partially Observable Settings](https://mlsanity.com/paper/2104.08492/) (2021)

## Built on by (most cited first, 2 of 2)

- [Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Study](https://mlsanity.com/paper/2609.07395/) (2026)
- [What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents](https://mlsanity.com/paper/2609.39989/) (2026)

## Abstract (arXiv)

Vision-language navigation agents achieve competitive average success on benchmark tasks, yet failures often arise through predictable trajectory-level breakdowns such as oscillation, stagnation, or inefficient detours. Reliable deployment, therefore, requires uncertainty signals that anticipate emerging failure dynamics during execution rather than reflect only instantaneous action entropy. We introduce GroundControl, a trajectory-consistent uncertainty estimator defined as statistical deviation from nominal goal-directed distance-to-goal dynamics aggregated over an episode. GroundControl models distance evolution using a constant-velocity Kalman filter and combines normalized innovation statistics with complementary trajectory features capturing progress, monotonicity, path efficiency, and oscillatory behavior. The resulting uncertainty score reflects geometric and temporal inconsistency in navigation behavior rather than local prediction dispersion. To evaluate uncertainty quality independently of task success, we formalize Selective Risk--Coverage Navigation (SRCN), a protocol that measures how effectively an uncertainty score ranks episodes by failure or inefficiency using risk--coverage curves and AURC / E-AURC summaries. Across five EB-Navigation splits (N=300 episodes), trajectory-consistent uncertainty achieves near-oracle ordering under success-based selective risk, with weighted-average mathrmE-AURCSR=0.0024 for the GPT-4o model, substantially outperforming entropy-, conformal-, and heuristic baselines. Under SPL-based selective evaluation, GroundControl consistently achieves the lowest AURC and E-AURC across models and navigation splits. These results show that modeling deviation from goal-directed dynamics provides an interpretable and robust signal for anticipating navigation failures in vision-language agents.