# Lana: A Language-Capable Navigator for Instruction Following and Generation

arXiv: https://arxiv.org/abs/2303.08409 · HTML: https://mlsanity.com/paper/2303.08409/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 99 references that are in the map.
- First posted on arXiv 15 Mar 2023.

## Builds on (most cited first, 30 of 60)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://mlsanity.com/paper/2108.10904/) (2021)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [VirTex: Learning Visual Representations from Textual Annotations](https://mlsanity.com/paper/2006.06666/) (2020)
- [REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments](https://mlsanity.com/paper/1904.10151/) (2019)
- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [A Recurrent Vision-and-Language BERT for Navigation](https://mlsanity.com/paper/2011.13922/) (2020)
- [History Aware Multimodal Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2110.13309/) (2021)
- [Vision-and-Dialog Navigation](https://mlsanity.com/paper/1907.04957/) (2019)
- [Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout](https://mlsanity.com/paper/1904.04195/) (2019)
- [Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training](https://mlsanity.com/paper/2002.10638/) (2020)
- [Scaling Up Vision-Language Pre-training for Image Captioning](https://mlsanity.com/paper/2111.12233/) (2021)
- [Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2202.11742/) (2022)
- [Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks](https://mlsanity.com/paper/1911.07883/) (2019)
- [Improving Vision-and-Language Navigation with Image-Text Pairs from the Web](https://mlsanity.com/paper/2004.14973/) (2020)
- [Self-Monitoring Navigation Agent via Auxiliary Progress Estimation](https://mlsanity.com/paper/1901.03035/) (2019)
- [Episodic Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2105.06453/) (2021)
- [Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation](https://mlsanity.com/paper/1905.12255/) (2019)

## Built on by (most cited first, 30 of 40)

- [ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2304.03047/) (2023)
- [DREAMWALKER: Mental Planning for Continuous Vision-Language Navigation](https://mlsanity.com/paper/2308.07498/) (2023)
- [PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation](https://mlsanity.com/paper/2305.19195/) (2023)
- [Bird's-Eye-View Scene Graph for Vision-Language Navigation](https://mlsanity.com/paper/2308.04758/) (2023)
- [Volumetric Environment Representation for Vision-Language Navigation](https://mlsanity.com/paper/2403.14158/) (2024)
- [Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigation](https://mlsanity.com/paper/2311.13209/) (2023)
- [Controllable Navigation Instruction Generation with Chain of Thought Prompting](https://mlsanity.com/paper/2407.07433/) (2024)
- [Navigation Instruction Generation with BEV Perception and Large Language Models](https://mlsanity.com/paper/2407.15087/) (2024)
- [Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheel](https://mlsanity.com/paper/2412.08467/) (2024)
- [Vision-Language Navigation with Energy-Based Policy](https://mlsanity.com/paper/2410.14250/) (2024)
- [NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLM](https://mlsanity.com/paper/2502.11142/) (2025)
- [FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Tasks](https://mlsanity.com/paper/2503.13966/) (2025)
- [Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigation](https://mlsanity.com/paper/2505.20897/) (2025)
- [Verifiably Following Complex Robot Instructions with Foundation Models](https://mlsanity.com/paper/2402.11498/) (2024)
- [COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigation](https://mlsanity.com/paper/2503.24065/) (2025)
- [DOPE: Dual Object Perception-Enhancement Network for Vision-and-Language Navigation](https://mlsanity.com/paper/2505.00743/) (2025)
- [NavHint: Vision and Language Navigation Agent with a Hint Generator](https://mlsanity.com/paper/2402.02559/) (2024)
- [Loc4Plan: Locating Before Planning for Outdoor Vision and Language Navigation](https://mlsanity.com/paper/2408.05090/) (2024)
- [Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environments](https://mlsanity.com/paper/2407.21452/) (2024)
- [Exploring What Why and How: A Multifaceted Benchmark for Causation Understanding of Video Anomaly](https://mlsanity.com/paper/2412.07183/) (2024)
- [Can LLMs Generate Human-Like Wayfinding Instructions? Towards Platform-Agnostic Embodied Instruction Synthesis](https://mlsanity.com/paper/2403.11487/) (2024)
- [GoViG: Goal-Conditioned Visual Navigation Instruction Generation via Multimodal Reasoning](https://mlsanity.com/paper/2508.09547/) (2025)
- [NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigation](https://mlsanity.com/paper/2510.16457/) (2025)
- [Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotations](https://mlsanity.com/paper/2506.08566/) (2025)
- [NavComposer: Composing Language Instructions for Navigation Trajectories through Action-Scene-Object Modularization](https://mlsanity.com/paper/2507.10894/) (2025)
- [Trajectory-Diversity-Driven Robust Vision-and-Language Navigation](https://mlsanity.com/paper/2603.15370/) (2026)
- [Planning with Logical Graph-based Language Model for Instruction Generation](https://mlsanity.com/paper/2308.13782/) (2023)
- [3D Gaussian Map with Open-Set Semantic Grouping for Vision-Language Navigation](https://mlsanity.com/paper/2605.26500/) (2026)
- [Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generation](https://mlsanity.com/paper/2409.05583/) (2024)
- [Think Hierarchically, Act Dynamically: Hierarchical Multi-modal Fusion and Reasoning for Vision-and-Language Navigation](https://mlsanity.com/paper/2504.16516/) (2025)

## Abstract (arXiv)

Recently, visual-language navigation (VLN) -- entailing robot agents to follow navigation instructions -- has shown great advance. However, existing literature put most emphasis on interpreting instructions into actions, only delivering "dumb" wayfinding agents. In this article, we devise LANA, a language-capable navigation agent which is able to not only execute human-written navigation commands, but also provide route descriptions to humans. This is achieved by simultaneously learning instruction following and generation with only one single model. More specifically, two encoders, respectively for route and language encoding, are built and shared by two decoders, respectively, for action prediction and instruction generation, so as to exploit cross-task knowledge and capture task-specific characteristics. Throughout pretraining and fine-tuning, both instruction following and generation are set as optimization objectives. We empirically verify that, compared with recent advanced task-specific solutions, LANA attains better performances on both instruction following and route description, with nearly half complexity. In addition, endowed with language generation capability, LANA can explain to humans its behaviors and assist human's wayfinding. This work is expected to foster future efforts towards building more trustworthy and socially-intelligent navigation robots.