# VQA-based Robotic State Recognition Optimized with Genetic Algorithm

arXiv: https://arxiv.org/abs/2303.05052 · HTML: https://mlsanity.com/paper/2303.05052/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2023.

## Built on by (most cited first, 10 of 10)

- [Real-World Robot Applications of Foundation Models: A Review](https://mlsanity.com/paper/2402.05741/) (2024)
- [Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviors](https://mlsanity.com/paper/2303.05674/) (2023)
- [Map-based Modular Approach for Zero-shot Embodied Question Answering](https://mlsanity.com/paper/2405.16559/) (2024)
- [Interactively Robot Action Planning with Uncertainty Analysis and Active Questioning by Large Language Model](https://mlsanity.com/paper/2308.15684/) (2023)
- [Semantic Scene Difference Detection in Daily Life Patroling by Mobile Robots using Pre-Trained Large-Scale Vision-Language Model](https://mlsanity.com/paper/2309.16552/) (2023)
- [Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-box Optimization](https://mlsanity.com/paper/2403.08239/) (2024)
- [Daily Assistive View Control Learning of Low-Cost Low-Rigidity Robot via Large-Scale Vision-Language Model](https://mlsanity.com/paper/2312.07451/) (2023)
- [Binary State Recognition by Robots using Visual Question Answering of Pre-Trained Vision-Language Model](https://mlsanity.com/paper/2310.16405/) (2023)
- [Robotic Environmental State Recognition with Pre-Trained Vision-Language Models and Black-Box Optimization](https://mlsanity.com/paper/2409.17519/) (2024)
- [Robotic State Recognition with Image-to-Text Retrieval Task of Pre-Trained Vision-Language Model and Black-Box Optimization](https://mlsanity.com/paper/2410.22707/) (2024)

## Abstract (arXiv)

State recognition of objects and environment in robots has been conducted in various ways. In most cases, this is executed by processing point clouds, learning images with annotations, and using specialized sensors. In contrast, in this study, we propose a state recognition method that applies Visual Question Answering (VQA) in a Pre-Trained Vision-Language Model (PTVLM) trained from a large-scale dataset. By using VQA, it is possible to intuitively describe robotic state recognition in the spoken language. On the other hand, there are various possible ways to ask about the same event, and the performance of state recognition differs depending on the question. Therefore, in order to improve the performance of state recognition using VQA, we search for an appropriate combination of questions using a genetic algorithm. We show that our system can recognize not only the open/closed of a refrigerator door and the on/off of a display, but also the open/closed of a transparent door and the state of water, which have been difficult to recognize.