mlsanityData to 7 Oct 2026

VQA-based Robotic State Recognition Optimized with Genetic Algorithm

arXiv 2303.05052 ↗First posted 9 Mar 2023cs.ROcited by 10

Not on the map. Why?

Abstract (arXiv)

State recognition of objects and environment in robots has been conducted in various ways. In most cases, this is executed by processing point clouds, learning images with annotations, and using specialized sensors. In contrast, in this study, we propose a state recognition method that applies Visual Question Answering (VQA) in a Pre-Trained Vision-Language Model (PTVLM) trained from a large-scale dataset. By using VQA, it is possible to intuitively describe robotic state recognition in the spoken language. On the other hand, there are various possible ways to ask about the same event, and the performance of state recognition differs depending on the question. Therefore, in order to improve the performance of state recognition using VQA, we search for an appropriate combination of questions using a genetic algorithm. We show that our system can recognize not only the open/closed of a refrigerator door and the on/off of a display, but also the open/closed of a transparent door and the state of water, which have been difficult to recognize.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Mar 2023.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 5.

10 papers.

  1. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  2. 2023Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviorscited by 12
  3. 2024Map-based Modular Approach for Zero-shot Embodied Question Answeringcited by 9
  4. 2023Interactively Robot Action Planning with Uncertainty Analysis and Active Questioning by Large Language Modelcited by 3
  5. 2023Semantic Scene Difference Detection in Daily Life Patroling by Mobile Robots using Pre-Trained Large-Scale Vision-Language Modelcited by 3
  6. 2024Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-box Optimizationcited by 3
  7. 2023Daily Assistive View Control Learning of Low-Cost Low-Rigidity Robot via Large-Scale Vision-Language Modelcited by 1
  8. 2023Binary State Recognition by Robots using Visual Question Answering of Pre-Trained Vision-Language Modelcited by 0
  9. 2024Robotic Environmental State Recognition with Pre-Trained Vision-Language Models and Black-Box Optimizationcited by 0
  10. 2024Robotic State Recognition with Image-to-Text Retrieval Task of Pre-Trained Vision-Language Model and Black-Box Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RoboVista: Evaluating Vision Language Models for Diverse Robot Applications
  2. 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering
  3. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  4. 2025Visual question answering: from early developments to recent advances -- a survey
  5. 2023Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts
  6. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
  7. 2024EfficientEQA: An Efficient Approach to Open-Vocabulary Embodied Question Answering
  8. 2025Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets