mlsanityData to 14 Oct 2026

Robotic State Recognition with Image-to-Text Retrieval Task of Pre-Trained Vision-Language Model and Black-Box Optimization

arXiv 2410.22707 ↗First posted 30 Oct 2024cs.RO cs.AI cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

State recognition of the environment and objects, such as the open/closed state of doors and the on/off of lights, is indispensable for robots that perform daily life support and security tasks. Until now, state recognition methods have been based on training neural networks from manual annotations, preparing special sensors for the recognition, or manually programming to extract features from point clouds or raw images. In contrast, we propose a robotic state recognition method using a pre-trained vision-language model, which is capable of Image-to-Text Retrieval (ITR) tasks. We prepare several kinds of language prompts in advance, calculate the similarity between these prompts and the current image by ITR, and perform state recognition. By applying the optimal weighting to each prompt using black-box optimization, state recognition can be performed with higher accuracy. Experiments show that this theory enables a variety of state recognitions by simply preparing multiple prompts without retraining neural networks or manual programming. In addition, since only prompts and their weights need to be prepared for each recognizer, there is no need to prepare multiple models, which facilitates resource management. It is possible to recognize the open/closed state of transparent doors, the state of whether water is running or not from a faucet, and even the qualitative state of whether a kitchen is clean or not, which have been challenging so far, through language.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 21 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

13 of 21 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Real-World Robot Applications of Foundation Models: A Review8 Feb
  2. 2023Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviors10 Mar

Other references · 11

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  4. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  5. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  7. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  8. 2023A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Modelscited by 39
  9. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  10. 2023VQA-based Robotic State Recognition Optimized with Genetic Algorithmcited by 10
  11. 2018Supervised Autonomous Locomotion and Manipulation for Disaster Response with a Centaur-like Robotcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Robotic Environmental State Recognition with Pre-Trained Vision-Language Models and Black-Box Optimization
  2. 2023Binary State Recognition by Robots using Visual Question Answering of Pre-Trained Vision-Language Model
  3. 2024Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-box Optimization
  4. 2024Details Make a Difference: Object State-Sensitive Neurorobotic Task Planning
  5. 2025KRAST: Knowledge-Augmented Robotic Action Recognition with Structured Text for Vision-Language Models
  6. 2025Image-based Geo-localization for Robotics: Are Black-box Vision-Language Models there yet?
  7. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Control
  8. 2022PLOT: Prompt Learning with Optimal Transport for Vision-Language Models