Accessible Instruction-Following Agent
On the map: LLM agents and tool use → Multimodal and embodied agents
Abstract (arXiv)
Humans can collaborate and complete tasks based on visual signals and instruction from the environment. Training such a robot is difficult especially due to the understanding of the instruction and the complicated environment. Previous instruction-following agents are biased to English-centric corpus, making it unrealizable to be applied to users that use multiple languages or even low-resource languages. Nevertheless, the instruction-following agents are pre-trained in a mode that assumes the user can observe the environment, which limits its accessibility. In this work, we're trying to generalize the success of instruction-following agents to non-English languages with little corpus resources, and improve its intractability and accessibility. We introduce UVLN (Universal Vision-Language Navigation), a novel machine-translation instructional augmented framework for cross-lingual vision-language navigation, with a novel composition of state-of-the-art large language model (GPT3) with the image caption model (BLIP). We first collect a multilanguage vision-language navigation dataset via machine translation. Then we extend the standard VLN training objectives to a multilingual setting via a cross-lingual language encoder. The alignment between different languages is captured through a shared vision and action context via a cross-modal transformer, which encodes the inputs of language instruction, visual observation, and action decision sequences. To improve the intractability, we connect our agent with the large language model that informs the situation and current state to the user and also explains the action decisions. Experiments over Room Across Room Dataset prove the effectiveness of our approach. And the qualitative results show the promising intractability and accessibility of our instruction-following agent.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 32 of its 41 references that are in the map. First posted on arXiv 8 May 2023.
Builds on
32 of 41 references.
References · 32
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
- 2020Beyond English-Centric Multilingual Machine Translationcited by 559
- 2018On Evaluation of Embodied Navigation Agentscited by 526
- 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
- 2020Semi-Supervised Semantic Segmentation with Cross-Consistency Trainingcited by 375
- 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
- 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
- 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
- 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
- 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
20 more
- 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
- 20213D Human Action Representation Learning via Cross-View Consistency Pursuitcited by 105
- 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 102
- 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 95
- 2021PixMatch: Unsupervised Domain Adaptation via Pixelwise Consistency Trainingcited by 80
- 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
- 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
- 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
- 2019Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigationcited by 54
- 2021Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigationcited by 47
- 2017Doubly-Attentive Decoder for Multi-modal Neural Machine Translationcited by 46
- 2017Incorporating Global Visual Features into Attention-Based Neural Machine Translationcited by 39
- 2021Adaptive Consistency Regularization for Semi-Supervised Transfer Learningcited by 37
- 2021Few-shot Open-set Recognition by Transformation Consistencycited by 29
- 2019MULE: Multimodal Universal Language Embeddingcited by 22
- 2019Counterfactual Vision-and-Language Navigation via Adversarial Path Samplingcited by 22
- 2020Learning to Scale Multilingual Representations for Vision-Language Taskscited by 22
- 2015Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learningcited by 18
- 2017Image Pivoting for Learning Multilingual Multimodal Representationscited by 17
- 2019Cross-Lingual Vision-Language Navigationcited by 11
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning
- 2018Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention
- 2026Beyond English: Uncovering the Multilingual Gap in Vision-Language-Action Models
- 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Tasks
- 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning
- 2025Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents
- 2026Sim-to-Real Transfer of Vision-Language Navigation in Continuous Environments Using an Ackermann-Steered Mobile Robot
- 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning