mlsanityData to 14 Oct 2026

LOC-ZSON: Language-driven Object-Centric Zero-Shot Object Retrieval and Navigation

arXiv 2405.05363 ↗First posted 8 May 2024cs.CV cs.ROcited by 8

Not on the map. Why?

Abstract (arXiv)

In this paper, we present LOC-ZSON, a novel Language-driven Object-Centric image representation for object navigation task within complex scenes. We propose an object-centric image representation and corresponding losses for visual-language model (VLM) fine-tuning, which can handle complex object-level queries. In addition, we design a novel LLM-based augmentation and prompt templates for stability during training and zero-shot inference. We implement our method on Astro robot and deploy it in both simulated and real-world environments for zero-shot object navigation. We show that our proposed method can achieve an improvement of 1.38 - 13.38% in terms of text-to-image recall on different benchmark settings for the retrieval task. For object navigation, we show the benefit of our approach in simulation and real world, showing 5% and 16.67% improvement in terms of navigation success rate, respectively.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 30 of its 35 references that are in the map. First posted on arXiv 8 May 2024.

Builds on

30 of 35 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  10. 2020Object-Centric Learning with Slot Attentioncited by 876
  11. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  12. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
18 more
  1. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
  2. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 392
  3. 2022Visual Language Maps for Robot Navigationcited by 365
  4. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 360
  5. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  6. 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227
  7. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  8. 2020SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decompositioncited by 190
  9. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 178
  10. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 165
  11. 2021GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinementcited by 109
  12. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigationcited by 74
  13. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 70
  14. 2022Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigationcited by 67
  15. 2022A Survey on Active Simultaneous Localization and Mapping: State of the Art and New Frontierscited by 66
  16. 2022Zero-shot object goal visual navigationcited by 31
  17. 2022Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioningcited by 21
  18. 2024On the Vulnerability of LLM/VLM-Controlled Roboticscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 2.

8 papers.

  1. 2024AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Modelscited by 32
  2. 2025Can Large Vision Language Models Read Maps Like a Human?cited by 30
  3. 2024Is 'Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuningcited by 8
  4. 2025Weakly-supervised VLM-guided Partial Contrastive Learning for Visual Language Navigationcited by 4
  5. 2025Efficient and Generalizable Environmental Understanding for Visual Navigationcited by 1
  6. 2026AERR-Nav: Adaptive Exploration-Recovery-Reminiscing Strategy for Zero-Shot Object Navigationcited by 1
  7. 2024Improving Zero-Shot ObjectNav with Generative Communicationcited by 0
  8. 2026A Dataset for Dynamic Human Preferences for Vision Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Zero-shot Object Navigation with Vision-Language Models Reasoning
  2. 2025PanoNav: Mapless Zero-Shot Object Navigation with Panoramic Scene Parsing and Dynamic Memory
  3. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models
  4. 2025History-Augmented Vision-Language Models for Frontier-Based Zero-Shot Object Navigation
  5. 2026SFCo-Nav: Efficient Zero-Shot Visual Language Navigation via Collaboration of Slow LLM and Fast Attributed Graph Alignment
  6. 2025osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoning
  7. 20263DGSNav: Enhancing Vision-Language Model Reasoning for Object Navigation via Active 3D Gaussian Splatting
  8. 2024Balancing Performance and Efficiency in Zero-shot Robotic Navigation