mlsanityData to 14 Oct 2026

March in Chat: Interactive Prompting for Remote Embodied Referring Expression

arXiv 2308.10141 ↗First posted 20 Aug 2023cs.CVcited by 34

Not on the map. Why?

Abstract (arXiv)

Many Vision-and-Language Navigation (VLN) tasks have been proposed in recent years, from room-based to object-based and indoor to outdoor. The REVERIE (Remote Embodied Referring Expression) is interesting since it only provides high-level instructions to the agent, which are closer to human commands in practice. Nevertheless, this poses more challenges than other VLN tasks since it requires agents to infer a navigation plan only based on a short instruction. Large Language Models (LLMs) show great potential in robot action planning by providing proper prompts. Still, this strategy has not been explored under the REVERIE settings. There are several new challenges. For example, the LLM should be environment-aware so that the navigation plan can be adjusted based on the current visual observation. Moreover, the LLM planned actions should be adaptable to the much larger and more complex REVERIE environment. This paper proposes a March-in-Chat (MiC) model that can talk to the LLM on the fly and plan dynamically based on a newly proposed Room-and-Object Aware Scene Perceiver (ROASP). Our MiC model outperforms the previous state-of-the-art by large margins by SPL and RGSPL metrics on the REVERIE benchmark.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 34 of its 41 references that are in the map. First posted on arXiv 20 Aug 2023.

Builds on

34 of 41 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  6. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  8. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  9. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  11. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  12. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
22 more
  1. 2018On Evaluation of Embodied Navigation Agentscited by 547
  2. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  3. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  4. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  5. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  6. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  7. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  8. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  9. 2019Vision-and-Dialog Navigationcited by 250
  10. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 239
  11. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
  12. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 176
  13. 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
  14. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 130
  15. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 130
  16. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 119
  17. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  18. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 70
  19. 2020Sub-Instruction Aware Vision-and-Language Navigationcited by 66
  20. 2021The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigationcited by 61
  21. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  22. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 50

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 13; 2025: 15; 2026: 4.

34 papers.

Sort:

By area:

  1. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  2. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  3. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 98
  4. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 86
  5. 2024Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigationcited by 65
  6. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 63
  7. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 62
  8. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 44
  9. 2025SmartWay: Enhanced Waypoint Prediction and Backtracking for Zero-Shot Vision-and-Language Navigationcited by 29
  10. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  11. 2024MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chainscited by 25
  12. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 21
22 more
  1. 2025NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLMcited by 19
  2. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 17
  3. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 17
  4. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 15
  5. 2023Advances in Embodied Navigation Using Large Language Models: A Surveycited by 14
  6. 2024Can LLMs Generate Human-Like Wayfinding Instructions? Towards Platform-Agnostic Embodied Instruction Synthesiscited by 6
  7. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  8. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agentscited by 5
  9. 2024Self-Explainable Affordance Learning with Embodied Captioncited by 4
  10. 2025EvolveNav: Empowering LLM-Based Vision-Language Navigation via Self-Improving Embodied Reasoningcited by 4
  11. 2025Boosting Zero-Shot VLN via Abstract Obstacle Map-Based Waypoint Prediction with TopoGraph-and-VisitInfo-Aware Promptingcited by 4
  12. 2025Think Hierarchically, Act Dynamically: Hierarchical Multi-modal Fusion and Reasoning for Vision-and-Language Navigationcited by 3
  13. 2025Vision-and-Language Navigation with Analogical Textual Descriptions in LLMscited by 3
  14. 2025UNeMo: Collaborative Visual-Language Reasoning and Navigation via a Multimodal World Modelcited by 3
  15. 2026Ask When It Pays: Cost-Aware Open-Ended Interaction for Instance Goal Navigationcited by 3
  16. 2024To Ask or Not to Ask? Detecting Absence of Information in Vision and Language Navigationcited by 1
  17. 2024Temporal-Spatial Object Relations Modeling for Vision-and-Language Navigationcited by 0
  18. 2024AI-Gadget Kit: Integrating Swarm User Interfaces with LLM-driven Agents for Rich Tabletop Game Applicationscited by 0
  19. 2025Learning to Reason and Navigate: Parameter Efficient Action Planning with Large Language Modelscited by 0
  20. 2026Scene Representation using 360deg Saliency Graph and its Application in Vision-based Indoor Navigationcited by 0
  21. 2026Interpretable Zero-shot Referring Expression Comprehension with Query-driven Scene Graphscited by 0
  22. 2026Systematic Multi-Agent Vision-and-Language Navigation: Formulation, Benchmark, and Methodcited by 0