# Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Prediction

arXiv: https://arxiv.org/abs/1811.04179 · HTML: https://mlsanity.com/paper/1811.04179/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2018.

## Built on by (most cited first, 30 of 35)

- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2204.06252/) (2022)
- [Waypoint Models for Instruction-guided Navigation in Continuous Environments](https://mlsanity.com/paper/2110.02207/) (2021)
- [FILM: Following Instructions in Language with Modular Methods](https://mlsanity.com/paper/2110.07342/) (2021)
- [AerialVLN: Vision-and-Language Navigation for UAVs](https://mlsanity.com/paper/2308.06735/) (2023)
- [SOON: Scenario Oriented Object Navigation with Graph-based Exploration](https://mlsanity.com/paper/2103.17138/) (2021)
- [Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology](https://mlsanity.com/paper/2410.07087/) (2024)
- [A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution](https://mlsanity.com/paper/2107.05612/) (2021)
- [Cross-modal Map Learning for Vision and Language Navigation](https://mlsanity.com/paper/2203.05137/) (2022)
- [Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention](https://mlsanity.com/paper/1812.04155/) (2018)
- [Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion](https://mlsanity.com/paper/2108.04927/) (2021)
- [Visual Navigation with Spatial Attention](https://mlsanity.com/paper/2104.09807/) (2021)
- [Iterative Vision-and-Language Navigation](https://mlsanity.com/paper/2210.03087/) (2022)
- [Vision-Language Navigation: A Survey and Taxonomy](https://mlsanity.com/paper/2108.11544/) (2021)
- [Episodic Memory Question Answering](https://mlsanity.com/paper/2205.01652/) (2022)
- [Core Challenges in Embodied Vision-Language Planning](https://mlsanity.com/paper/2106.13948/) (2021)
- [Continual Learning for Instruction Following from Realtime Feedback](https://mlsanity.com/paper/2212.09710/) (2022)
- [Exploring Spatial Representation to Enhance LLM Reasoning in Aerial Vision-Language Navigation](https://mlsanity.com/paper/2410.08500/) (2024)
- [Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions](https://mlsanity.com/paper/2406.19236/) (2024)
- [NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigation](https://mlsanity.com/paper/2411.08579/) (2024)
- [Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks](https://mlsanity.com/paper/2210.04476/) (2022)
- [CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissection](https://mlsanity.com/paper/2410.07540/) (2024)
- [Trajectory Prediction with Linguistic Representations](https://mlsanity.com/paper/2110.09741/) (2021)
- [Aerial Vision-and-Language Navigation with Grid-based View Selection and Map Construction](https://mlsanity.com/paper/2503.11091/) (2025)
- [Memory-Centric Embodied Question Answering](https://mlsanity.com/paper/2505.13948/) (2025)
- [Don't Copy the Teacher: Data and Model Challenges in Embodied Dialogue](https://mlsanity.com/paper/2210.04443/) (2022)
- [Multitask Multimodal Prompted Training for Interactive Embodied Task Completion](https://mlsanity.com/paper/2311.04067/) (2023)
- [Vision-Language Navigation with Embodied Intelligence: A Survey](https://mlsanity.com/paper/2402.14304/) (2024)
- [HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactions](https://mlsanity.com/paper/2503.14229/) (2025)

## Abstract (arXiv)

We propose an approach for mapping natural language instructions and raw observations to continuous control of a quadcopter drone. Our model predicts interpretable position-visitation distributions indicating where the agent should go during execution and where it should stop, and uses the predicted distributions to select the actions to execute. This two-step model decomposition allows for simple and efficient training using a combination of supervised learning and imitation learning. We evaluate our approach with a realistic drone simulator, and demonstrate absolute task-completion accuracy improvements of 16.85% over two state-of-the-art instruction-following methods.