Binary State Recognition by Robots using Visual Question Answering of Pre-Trained Vision-Language Model
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
Recognition of the current state is indispensable for the operation of a robot. There are various states to be recognized, such as whether an elevator door is open or closed, whether an object has been grasped correctly, and whether the TV is turned on or off. Until now, these states have been recognized by programmatically describing the state of a point cloud or raw image, by annotating and learning images, by using special sensors, etc. In contrast to these methods, we apply Visual Question Answering (VQA) from a Pre-Trained Vision-Language Model (PTVLM) trained on a large-scale dataset, to such binary state recognition. This idea allows us to intuitively describe state recognition in language without any re-training, thereby improving the recognition ability of robots in a simple and general way. We summarize various techniques in questioning methods and image processing, and clarify their properties through experiments.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 13 references that are in the map. First posted on arXiv 25 Oct 2023.
Builds on
7 of 13 references.
References · 7
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2015VQA: Visual Question Answeringcited by 2,489
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
- 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
- 2017Embodied Question Answeringcited by 335
- 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
- 2023VQA-based Robotic State Recognition Optimized with Genetic Algorithmcited by 10
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Robotic Environmental State Recognition with Pre-Trained Vision-Language Models and Black-Box Optimization
- 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
- 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
- 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering
- 2025Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets
- 2026RoboVista: Evaluating Vision Language Models for Diverse Robot Applications
- 2024Robotic State Recognition with Image-to-Text Retrieval Task of Pre-Trained Vision-Language Model and Black-Box Optimization
- 2023Expanding Frozen Vision-Language Models without Retraining: Towards Improved Robot Perception