# Speaking the Navigator's Language: Trajectory-Grounded Instruction Translation for Frozen Aerial VLN Agents

arXiv: https://arxiv.org/abs/2610.10635 · HTML: https://mlsanity.com/paper/2610.10635/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 40 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 10 of 10)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [A Survey on Vision-Language-Action Models for Embodied AI](https://mlsanity.com/paper/2405.14093/) (2024)
- [Openfly: A comprehensive platform for aerial vision-language navigation](https://mlsanity.com/paper/2502.18041/) (2025)
- [NORA-1.5: A Vision-Language-Action Model Trained using World Model- and Action-based Preference Rewards](https://mlsanity.com/paper/2511.14659/) (2025)
- [AeroVerse: UAV-Agent Benchmark Suite for Simulating, Pre-training, Finetuning, and Evaluating Aerospace Embodied World Models](https://mlsanity.com/paper/2408.15511/) (2024)
- [VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation](https://mlsanity.com/paper/2512.19021/) (2025)
- [Advances in Embodied Navigation Using Large Language Models: A Survey](https://mlsanity.com/paper/2311.00530/) (2023)
- [Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigation](https://mlsanity.com/paper/2503.18065/) (2025)
- [T-araVLN: Translator for Agricultural Robotic Agents on Vision-and-Language Navigation](https://mlsanity.com/paper/2509.06644/) (2025)

## Abstract (arXiv)

Aerial vision-and-language navigation (VLN) agents are typically trained on detail-rich, trajectory-aligned commands, whereas users issue short, intent-driven instructions; on a frozen OpenFly navigator, this instruction gap drops success rate (SR) from 31.03% to 11.33%. To scale translator training, we prompt a language model with human-written style examples to convert original commands into paired, intent-centered Weak commands, which yield 15.27% SR. We introduce the Trajectory-Grounded Instruction Translator (TGIT), a front-end that keeps the navigator frozen and translates Weak inputs into agent-executable commands by learning from its trajectory outcomes. The resulting Weak-trained translator raises Weak-input SR to 37.93% and transfers zero-shot to real human instructions (11.33%rightarrow32.51%); it also improves held-out OpenFly (4.95%rightarrow20.79%) and yields recovery on CityNav and AirVLN.