mlsanityData to 7 Oct 2026

Lana: A Language-Capable Navigator for Instruction Following and Generation

arXiv 2303.08409 ↗First posted 15 Mar 2023cs.CV cs.MM cs.ROcited by 39

Not on the map. Why?

Abstract (arXiv)

Recently, visual-language navigation (VLN) -- entailing robot agents to follow navigation instructions -- has shown great advance. However, existing literature put most emphasis on interpreting instructions into actions, only delivering "dumb" wayfinding agents. In this article, we devise LANA, a language-capable navigation agent which is able to not only execute human-written navigation commands, but also provide route descriptions to humans. This is achieved by simultaneously learning instruction following and generation with only one single model. More specifically, two encoders, respectively for route and language encoding, are built and shared by two decoders, respectively, for action prediction and instruction generation, so as to exploit cross-task knowledge and capture task-specific characteristics. Throughout pretraining and fine-tuning, both instruction following and generation are set as optimization objectives. We empirically verify that, compared with recent advanced task-specific solutions, LANA attains better performances on both instruction following and route description, with nearly half complexity. In addition, endowed with language generation capability, LANA can explain to humans its behaviors and assist human's wayfinding. This work is expected to foster future efforts towards building more trustworthy and socially-intelligent navigation robots.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 60 of its 99 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

60 of 99 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  12. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
38 more
  1. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  2. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  3. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  4. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  5. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  6. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  7. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  8. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  9. 2019Vision-and-Dialog Navigationcited by 246
  10. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  11. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  12. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  13. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  14. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  15. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  16. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  17. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  18. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  19. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  20. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  21. 2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
  22. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  23. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  24. 2022CenterCLIP: Token Clustering for Efficient Text-Video Retrievalcited by 86
  25. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  26. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  27. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  28. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  29. 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigationcited by 71
  30. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  31. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  32. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  33. 2017Unified Pragmatic Models for Generating and Following Instructionscited by 55
  34. 2020Active Visual Information Gathering for Vision-Language Navigationcited by 55
  35. 2015Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequencescited by 50
  36. 2022Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigationcited by 50
  37. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  38. 2022Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Modelscited by 48

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 13; 2025: 12; 2026: 8.

39 papers.

Sort:

By area:

  1. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  2. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  3. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  4. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  5. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  6. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  7. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  8. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 20
  9. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  10. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  11. 2025NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLMcited by 18
  12. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
27 more
  1. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 16
  2. 2024Verifiably Following Complex Robot Instructions with Foundation Modelscited by 15
  3. 2025DOPE: Dual Object Perception-Enhancement Network for Vision-and-Language Navigationcited by 13
  4. 2024NavHint: Vision and Language Navigation Agent with a Hint Generatorcited by 12
  5. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 12
  6. 2024Loc4Plan: Locating Before Planning for Outdoor Vision and Language Navigationcited by 11
  7. 2024Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environmentscited by 10
  8. 2024Exploring What Why and How: A Multifaceted Benchmark for Causation Understanding of Video Anomalycited by 9
  9. 2024Can LLMs Generate Human-Like Wayfinding Instructions? Towards Platform-Agnostic Embodied Instruction Synthesiscited by 6
  10. 2025GoViG: Goal-Conditioned Visual Navigation Instruction Generation via Multimodal Reasoningcited by 6
  11. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  12. 2025NavComposer: Composing Language Instructions for Navigation Trajectories through Action-Scene-Object Modularizationcited by 5
  13. 2023Planning with Logical Graph-based Language Model for Instruction Generationcited by 4
  14. 2025Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotationscited by 4
  15. 2026Trajectory-Diversity-Driven Robust Vision-and-Language Navigationcited by 4
  16. 20263D Gaussian Map with Open-Set Semantic Grouping for Vision-Language Navigationcited by 4
  17. 2024Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generationcited by 3
  18. 2025Think Hierarchically, Act Dynamically: Hierarchical Multi-modal Fusion and Reasoning for Vision-and-Language Navigationcited by 3
  19. 2025DialNav: Multi-turn Dialog Navigation with a Remote Guidecited by 3
  20. 2026VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructionscited by 2
  21. 2024To Ask or Not to Ask? Detecting Absence of Information in Vision and Language Navigationcited by 1
  22. 2025Fine-Grained Instruction-Guided Graph Reasoning for Vision-and-Language Navigationcited by 1
  23. 2026Uncertainty-Aware Gaussian Map for Vision-Language Navigationcited by 1
  24. 2026A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigationcited by 1
  25. 2026Beyond Textual Knowledge-Leveraging Multimodal Knowledge Bases for Enhancing Vision-and-Language Navigationcited by 0
  26. 2026Goal-oriented Navigation Instruction Generation with Tour Video Priorscited by 0
  27. 2026Time-Aware Assistive Navigationcited by 0