# Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation

arXiv: https://arxiv.org/abs/2104.10674 · HTML: https://mlsanity.com/paper/2104.10674/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2021.

## Built on by (most cited first, 30 of 47)

- [NaVILA: Legged Robot Vision-Language-Action Model for Navigation](https://mlsanity.com/paper/2412.04453/) (2024)
- [Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2202.11742/) (2022)
- [Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.02764/) (2022)
- [Waypoint Models for Instruction-guided Navigation in Continuous Environments](https://mlsanity.com/paper/2110.02207/) (2021)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)
- [LanguageRefer: Spatial-Language Model for 3D Visual Grounding](https://mlsanity.com/paper/2107.03438/) (2021)
- [Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2204.09667/) (2022)
- [Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models](https://mlsanity.com/paper/2407.07035/) (2024)
- [ShAPO: Implicit Representations for Multi-Object Shape, Appearance, and Pose Optimization](https://mlsanity.com/paper/2207.13691/) (2022)
- [Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigation](https://mlsanity.com/paper/2407.05890/) (2024)
- [A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility](https://mlsanity.com/paper/2202.02312/) (2022)
- [Learning from Unlabeled 3D Environments for Vision-and-Language Navigation](https://mlsanity.com/paper/2208.11781/) (2022)
- [SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2108.11945/) (2021)
- [Iterative Vision-and-Language Navigation](https://mlsanity.com/paper/2210.03087/) (2022)
- [UAV-Flow Colosseo: A Real-World Benchmark for Flying-on-a-Word UAV Imitation Learning](https://mlsanity.com/paper/2505.15725/) (2025)
- [Core Challenges in Embodied Vision-Language Planning](https://mlsanity.com/paper/2106.13948/) (2021)
- [Neural Fields in Robotics: A Survey](https://mlsanity.com/paper/2410.20220/) (2024)
- [Ground then Navigate: Language-guided Navigation in Dynamic Scenes](https://mlsanity.com/paper/2209.11972/) (2022)
- [FSD: Fast Self-Supervised Single RGB-D to Categorical 3D Objects](https://mlsanity.com/paper/2310.12974/) (2023)
- [VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation](https://mlsanity.com/paper/2512.19021/) (2025)
- [Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Grounding](https://mlsanity.com/paper/2303.04077/) (2023)
- [A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AI](https://mlsanity.com/paper/2505.01458/) (2025)
- [ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigation](https://mlsanity.com/paper/2509.12618/) (2025)
- [EmbodiedSplat: Personalized Real-to-Sim-to-Real Navigation with Gaussian Splats from a Mobile Device](https://mlsanity.com/paper/2509.17430/) (2025)
- [MLANet: Multi-Level Attention Network with Sub-instruction for Continuous Vision-and-Language Navigation](https://mlsanity.com/paper/2303.01396/) (2023)
- [Vision and Language Navigation in the Real World via Online Visual Language Mapping](https://mlsanity.com/paper/2310.10822/) (2023)
- [Ground-level Viewpoint Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2502.19024/) (2025)
- [Focus on Impact: Indoor Exploration with Intrinsic Motivation](https://mlsanity.com/paper/2109.08521/) (2021)
- [Vision-Language Navigation with Embodied Intelligence: A Survey](https://mlsanity.com/paper/2402.14304/) (2024)
- [DOROTHIE: Spoken Dialogue for Handling Unexpected Situations in Interactive Autonomous Driving Agents](https://mlsanity.com/paper/2210.12511/) (2022)

## Abstract (arXiv)

Deep Learning has revolutionized our ability to solve complex problems such as Vision-and-Language Navigation (VLN). This task requires the agent to navigate to a goal purely based on visual sensory inputs given natural language instructions. However, prior works formulate the problem as a navigation graph with a discrete action space. In this work, we lift the agent off the navigation graph and propose a more complex VLN setting in continuous 3D reconstructed environments. Our proposed setting, Robo-VLN, more closely mimics the challenges of real world navigation. Robo-VLN tasks have longer trajectory lengths, continuous action spaces, and challenges such as obstacles. We provide a suite of baselines inspired by state-of-the-art works in discrete VLN and show that they are less effective at this task. We further propose that decomposing the task into specialized high- and low-level policies can more effectively tackle this task. With extensive experiments, we show that by using layered decision making, modularized training, and decoupling reasoning and imitation, our proposed Hierarchical Cross-Modal (HCM) agent outperforms existing baselines in all key metrics and sets a new benchmark for Robo-VLN.