mlsanityData to 14 Oct 2026

Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigation

arXiv 2411.07848 ↗First posted 12 Nov 2024cs.RO cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Large scale scenes such as multifloor homes can be robustly and efficiently mapped with a 3D graph of landmarks estimated jointly with robot poses in a factor graph, a technique commonly used in commercial robots such as drones and robot vacuums. In this work, we propose Language-Inferred Factor Graph for Instruction Following (LIFGIF), a zero-shot method to ground natural language instructions in such a map. LIFGIF also includes a policy for following natural language navigation instructions in a novel environment while the map is constructed, enabling robust navigation performance in the physical world. To evaluate LIFGIF, we present a new dataset, Object-Centric VLN (OC-VLN), in order to evaluate grounding of object-centric natural language navigation instructions. We compare to two state-of-the-art zero-shot baselines from related tasks, Object Goal Navigation and Vision Language Navigation, to demonstrate that LIFGIF outperforms them across all our evaluation metrics on OCVLN. Finally, we successfully demonstrate the effectiveness of LIFGIF for performing zero-shot object-centric instruction following in the real world on a Boston Dynamics Spot robot.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 71 of its 81 references that are in the map. First posted on arXiv 12 Nov 2024.

Builds on

71 of 81 references.

References · 71

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023Segment Anythingcited by 8,234
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  9. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  11. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 982
  12. 2020ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAMcited by 878
38 more
  1. 2016ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Camerascited by 806
  2. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  3. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  4. 2018On Evaluation of Embodied Navigation Agentscited by 547
  5. 2023LERF: Language Embedded Radiance Fieldscited by 543
  6. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  7. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
  8. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  9. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  10. 2022Visual Language Maps for Robot Navigationcited by 365
  11. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  12. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  13. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  14. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  15. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  16. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  17. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 244
  18. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 208
  19. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 205
  20. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 196
  21. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
  22. 2019Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mappingcited by 184
  23. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  24. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  25. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  26. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  27. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 157
  28. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  29. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  30. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 129
  31. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 124
  32. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 122
  33. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 119
  34. 2022Habitat-Matterport 3D Semantics Datasetcited by 110
  35. 2022Navigating to Objects in the Real Worldcited by 106
  36. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 105
  37. 2020MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigationcited by 95
  38. 2021Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environmentscited by 94

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2026GroundControl: Anticipating Navigation Failures in Vision-Language Agents via Trajectory-Consistent Uncertainty Estimatescited by 2
  2. 2024AdaVLN: Towards Visual Language Navigation in Continuous Indoor Environments with Moving Humanscited by 1
  3. 2025MLFM: Multi-Layered Feature Maps for Richer Language Understanding in Zero-Shot Semantic Navigationcited by 1
  4. 2026Expanding Spatial and Temporal Context for Robotic Imitation Learning With Scene Graphscited by 0